{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# pytorch 基础知识"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 神经网络层管理工具`nn.ModuleList` 和 `nn.Sequential`\n",
    "\n",
    "https://blog.51cto.com/u_13977270/3398081\n",
    "\n",
    "`nn.ModuleList` 和 `nn.Sequential` 都是 PyTorch 中用于管理神经网络层的工具，但是它们之间存在一些关键区别。下面我将通过具体的例子来说明这两种工具的不同之处。\n",
    "\n",
    "### 示例代码\n",
    "\n",
    "#### 使用 `nn.Sequential`\n",
    "\n",
    "首先，我们来看一下使用 `nn.Sequential` 的例子：\n",
    "\n",
    "```python\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class SequentialModel(nn.Module):\n",
    "    def __init__(self, in_features=10, out_features=2):\n",
    "        super(SequentialModel, self).__init__()\n",
    "        # 使用Sequential来定义网络层\n",
    "        self.model = nn.Sequential(\n",
    "            nn.Linear(in_features, 100, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(100, 30, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(30, out_features, bias=True)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "\n",
    "# 实例化模型\n",
    "seq_model = SequentialModel()\n",
    "print(seq_model)\n",
    "```\n",
    "\n",
    "#### 使用 `nn.ModuleList`\n",
    "\n",
    "接着，我们来看一下使用 `nn.ModuleList` 的例子：\n",
    "\n",
    "```python\n",
    "class ModuleListModel(nn.Module):\n",
    "    def __init__(self, in_features=10, out_features=2):\n",
    "        super(ModuleListModel, self).__init__()\n",
    "        # 使用ModuleList来定义网络层\n",
    "        self.layers = nn.ModuleList([\n",
    "            nn.Linear(in_features, 100, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(100, 30, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(30, out_features, bias=True)\n",
    "        ])\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 依次通过每一层进行前向传播\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n",
    "\n",
    "# 实例化模型\n",
    "module_list_model = ModuleListModel()\n",
    "print(module_list_model)\n",
    "```\n",
    "\n",
    "### 区别总结\n",
    "\n",
    "1. **前向传播**：\n",
    "   - `nn.Sequential`：前向传播时，只需要将输入传递给 `model`，剩下的工作（按顺序应用每一层）将自动完成。\n",
    "   - `nn.ModuleList`：需要显式地遍历每一层，并手动调用每一层的前向传播方法。\n",
    "\n",
    "2. **灵活性**：\n",
    "   - `nn.Sequential`：适用于结构相对固定、顺序明确的网络。\n",
    "   - `nn.ModuleList`：更适合需要动态构建或修改网络的情况，如条件分支、循环等。\n",
    "\n",
    "### 示例运行结果\n",
    "\n",
    "当你运行上述代码时，你会看到两个模型的定义略有不同：\n",
    "\n",
    "- 使用 `nn.Sequential` 的模型看起来像是一个固定的顺序容器。\n",
    "- 使用 `nn.ModuleList` 的模型则更像是一个可以自由操作的列表。\n",
    "\n",
    "根据你的具体需求，你可以选择最适合的方式来定义你的模型。如果你的模型结构非常规整并且不需要频繁改动，那么 `nn.Sequential` 会更加方便；如果你需要更高的灵活性，那么 `nn.ModuleList` 会是更好的选择。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch import optim\n",
    "from torch.nn import functional as F\n",
    "from torch.utils.data import TensorDataset\n",
    "from torch.utils.data import DataLoader\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.3403, 0.8266, 0.9849],\n",
      "        [0.0031, 0.8972, 0.1040],\n",
      "        [0.6214, 0.7369, 0.9363],\n",
      "        [0.6827, 0.2049, 0.8847]])\n"
     ]
    }
   ],
   "source": [
    "# 生成一个随机4,3的张量\n",
    "x=torch.rand(4,3)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_s = nn.Sequential(\n",
    "            nn.Linear(3, 100, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(100, 30, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(30, 1, bias=True)\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.0546],\n",
       "        [-0.0128],\n",
       "        [-0.0905],\n",
       "        [-0.1036]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_s(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_m = nn.ModuleList([\n",
    "    nn.Linear(3, 100, bias=True),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(100, 30, bias=True),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(30, 1, bias=True)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "out=model_m[0](x)\n",
    "out=model_m[1](out)\n",
    "out=model_m[2](out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_l = [\n",
    "    nn.Linear(3, 100, bias=True),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(100, 30, bias=True),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(30, 1, bias=True)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "out=model_l[0](x)\n",
    "out=model_l[1](out)\n",
    "out=model_l[2](out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method Module.parameters of ModuleList(\n",
       "  (0): Linear(in_features=3, out_features=100, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=100, out_features=30, bias=True)\n",
       "  (3): ReLU()\n",
       "  (4): Linear(in_features=30, out_features=1, bias=True)\n",
       ")>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_m.parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### classwork 1\n",
    "\n",
    "* 把下面神经网络改为通过`nn.ModuleList`和`nn.Sequential`来构建，并完成训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "class Model0(nn.Module):\n",
    "    def __init__(self,in_features=10,out_features=2):\n",
    "        super(Model0,self).__init__()\n",
    "        self.h1=nn.Linear(in_features,100,bias=True)\n",
    "        self.h2=nn.Linear(100,30,bias=True)\n",
    "        self.out=nn.Linear(30,out_features,bias=True)\n",
    "\n",
    "    def forward(self, x):\n",
    "        h1_out=self.h1(x)\n",
    "        h1_out_r=torch.relu(h1_out)\n",
    "        h2_out=self.h2(h1_out_r)\n",
    "        h2_out_r=torch.relu(h2_out)\n",
    "        out_out=self.out(h2_out_r)\n",
    "        return out_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModelSequential(nn.Module):\n",
    "    def __init__(self, in_features=10, out_features=2):\n",
    "        super(ModelSequential, self).__init__()\n",
    "        # 定义每一层及其激活函数\n",
    "        self.model = nn.Sequential(\n",
    "            nn.Linear(in_features, 100, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(100, 30, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(30, out_features, bias=True)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 由于所有的层和激活函数都已经被封装在Sequential中，\n",
    "        # 因此这里的forward只需要直接传递输入即可。\n",
    "        return self.model(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class ModelModuleList(nn.Module):\n",
    "    def __init__(self, in_features=10, out_features=2):\n",
    "        super(ModelModuleList, self).__init__()\n",
    "        # 使用ModuleList来存储网络层\n",
    "        self.layers = nn.ModuleList([\n",
    "            nn.Linear(in_features, 100, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(100, 30, bias=True),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(30, out_features, bias=True)\n",
    "        ])\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 依次通过每一层进行前向传播\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.6942, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6914, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6977, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6939, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6951, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6891, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6935, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6946, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6937, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.7000, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6949, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6898, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6940, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6933, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6954, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6874, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6927, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6940, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6941, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.7002, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6945, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6896, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6932, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6934, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6949, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6873, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6923, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6937, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6941, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6999, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6941, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6896, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6930, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6933, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6943, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6875, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6919, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6935, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6940, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6994, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6936, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6895, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6928, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6933, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6937, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6878, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6916, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6934, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6939, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n",
      "tensor(0.6986, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)\n"
     ]
    }
   ],
   "source": [
    "lr=0.1\n",
    "gamma=0.5\n",
    "epochs=5\n",
    "\n",
    "torch.manual_seed(40)\n",
    "x = torch.rand((500,20),dtype=torch.float32)\n",
    "y = torch.randint(low=0,high=2,size=(500,1),dtype=torch.float32)\n",
    "dataset = TensorDataset(x, y)\n",
    "dataloader = DataLoader(dataset, batch_size=50) \n",
    "\n",
    "net=ModelModuleList(20,1)\n",
    "#print(output)\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "#criterion = nn.MSELoss()\n",
    "opt=optim.SGD(net.parameters() , lr=lr , momentum = gamma)\n",
    "\n",
    "for i in range(epochs):\n",
    "    for x1,y1 in dataloader:\n",
    "        output=net.forward(x1)\n",
    "        loss = criterion(output,y1)#在PyTorch中,有些情况下目标向量(target)需要使用.long()方法转换为长整型tensor(long tensor),\n",
    "        opt.zero_grad()\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        print(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 张量的连续性、contiguous函数\n",
    "\n",
    "https://blog.csdn.net/m0_48241022/article/details/132804698\n",
    "\n",
    "在pytorch中，tensor的实际数据以一维数组（storage）的形式存储于某个连续的内存中，以“行优先”进行存储\n",
    " tensor连续（contiguous）是指tensor的storage元素排列顺序与其按行优先时的元素排列顺序相同\n",
    "\n",
    "\n",
    " tensor不连续会导致某些操作无法进行，比如view()就无法进行。在上面的例子中：由于 b 是不连续的，所以对其进行view()操作会报错；b.view(3,3)没报错，因为b本身的shape就是(3,3)。\n",
    "\n",
    "  tensor.contiguous()返回一个与原始tensor有相同元素的 “连续”tensor，如果原始tensor本身就是连续的，则返回原始tensor。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"37.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(url= \"37.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6],\n",
      "        [7, 8, 9]])\n",
      " 1\n",
      " 2\n",
      " 3\n",
      " 4\n",
      " 5\n",
      " 6\n",
      " 7\n",
      " 8\n",
      " 9\n",
      "[torch.storage.TypedStorage(dtype=torch.int64, device=cpu) of size 9]\n",
      "True\n",
      "tensor([[1, 4, 7],\n",
      "        [2, 5, 8],\n",
      "        [3, 6, 9]])\n",
      " 1\n",
      " 2\n",
      " 3\n",
      " 4\n",
      " 5\n",
      " 6\n",
      " 7\n",
      " 8\n",
      " 9\n",
      "[torch.storage.TypedStorage(dtype=torch.int64, device=cpu) of size 9]\n",
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "a = torch.tensor([[1, 2, 3],\n",
    "                  [4, 5, 6],\n",
    "                  [7, 8, 9]])\n",
    "print(a)\n",
    "print(a.storage())\n",
    "print(a.is_contiguous())  # a是连续的\n",
    "\n",
    " \n",
    "b = a.t()  # b是a的转置\n",
    "print(b)\n",
    "print(b.storage())\n",
    "print(b.is_contiguous())  # b是不连续的\n",
    "\n",
    "#让张量变连续\n",
    "c = b.contiguous()\n",
    "print(c.is_contiguous())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 常用的张量维度变换函数\n",
    "\n",
    "### `view()`\n",
    "\n",
    "`view()` 函数可以用来重塑张量，使其具有新的形状。新形状必须与原始张量的元素数量相匹配。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Tensor:\n",
      "tensor([[ 0.4461,  0.6512,  1.4922],\n",
      "        [ 1.9560,  0.8955,  0.1616],\n",
      "        [ 1.0333, -0.5607,  1.2151],\n",
      "        [ 0.9077, -0.2138,  1.4773]])\n",
      "\n",
      "Reshaped Tensor using view():\n",
      "tensor([[ 0.4461,  0.6512],\n",
      "        [ 1.4922,  1.9560],\n",
      "        [ 0.8955,  0.1616],\n",
      "        [ 1.0333, -0.5607],\n",
      "        [ 1.2151,  0.9077],\n",
      "        [-0.2138,  1.4773]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(4, 3)\n",
    "print(\"Original Tensor:\")\n",
    "print(x)\n",
    "\n",
    "reshaped_x = x.view(-1, 2)  # 使用-1自动推断该维度大小\n",
    "print(\"\\nReshaped Tensor using view():\")\n",
    "print(reshaped_x)\n",
    "\n",
    "# 注意事项：使用 view() 时，新形状的总元素数必须与原张量相同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = torch.randint(low=0,high=3,size=(500,1),dtype=torch.float32)\n",
    "#y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#y.view(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `transpose()` / `t()`\n",
    "\n",
    "`transpose()` 函数用于交换两个维度，而 `t()` 是 `transpose(0, 1)` 的简写，用于交换两个相邻的维度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.4461,  1.9560,  1.0333,  0.9077],\n",
      "        [ 0.6512,  0.8955, -0.5607, -0.2138],\n",
      "        [ 1.4922,  0.1616,  1.2151,  1.4773]])\n",
      "tensor([[ 0.4461,  1.9560,  1.0333,  0.9077],\n",
      "        [ 0.6512,  0.8955, -0.5607, -0.2138],\n",
      "        [ 1.4922,  0.1616,  1.2151,  1.4773]])\n"
     ]
    }
   ],
   "source": [
    "transposed_x = x.transpose(0, 1)  # 交换第一个和第二个维度\n",
    "print(transposed_x)\n",
    "\n",
    "transposed_x_t = x.t()  # 只适用于二维张量\n",
    "print(transposed_x_t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `unsqueeze()` 与 `squeeze()`\n",
    "\n",
    "`unsqueeze()` 函数用于在张量中增加一个新的维度（即增加一个轴）。\n",
    "\n",
    "`squeeze()` 函数用于删除单维度条目（即大小为1的维度）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[ 0.4461,  0.6512,  1.4922],\n",
      "         [ 1.9560,  0.8955,  0.1616],\n",
      "         [ 1.0333, -0.5607,  1.2151],\n",
      "         [ 0.9077, -0.2138,  1.4773]]])\n",
      "torch.Size([1, 4, 3])\n"
     ]
    }
   ],
   "source": [
    "unsqueezed_x = x.unsqueeze(0)  # 在第一个维度增加一个轴\n",
    "print(unsqueezed_x)\n",
    "print(unsqueezed_x.shape)\n",
    "# 注意事项：可以在任何位置增加新的维度，但增加的维度大小始终为1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.4461,  0.6512,  1.4922],\n",
      "        [ 1.9560,  0.8955,  0.1616],\n",
      "        [ 1.0333, -0.5607,  1.2151],\n",
      "        [ 0.9077, -0.2138,  1.4773]])\n",
      "torch.Size([4, 3])\n"
     ]
    }
   ],
   "source": [
    "squeezed_x = unsqueezed_x.squeeze(0)  # 删除第一个维度\n",
    "print(squeezed_x)\n",
    "print(squeezed_x.shape)\n",
    "\n",
    "# 注意事项：可以指定某个维度删除，如果不指定，则删除所有大小为1的维度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([500, 1])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 2., 1., 2., 0., 0., 1., 2., 1., 1., 2., 1., 2., 0., 1., 0., 0., 2.,\n",
       "        0., 2., 1., 2., 2., 1., 2., 2., 0., 1., 2., 2., 0., 2., 0., 1., 0., 1.,\n",
       "        1., 0., 2., 1., 1., 1., 2., 0., 0., 1., 1., 0., 0., 0., 2., 2., 1., 0.,\n",
       "        1., 1., 2., 0., 0., 1., 0., 2., 2., 2., 0., 0., 0., 1., 2., 2., 2., 2.,\n",
       "        2., 2., 0., 2., 2., 2., 1., 1., 0., 0., 2., 2., 2., 2., 2., 1., 2., 1.,\n",
       "        2., 0., 2., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 2., 2., 0., 0., 0.,\n",
       "        1., 0., 2., 0., 1., 1., 2., 0., 2., 1., 1., 0., 2., 2., 2., 2., 0., 0.,\n",
       "        2., 2., 1., 0., 0., 2., 0., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 0.,\n",
       "        0., 0., 0., 2., 1., 2., 2., 1., 0., 0., 1., 1., 0., 1., 2., 0., 2., 2.,\n",
       "        0., 2., 1., 0., 1., 2., 2., 2., 1., 1., 0., 2., 1., 2., 0., 2., 0., 2.,\n",
       "        0., 0., 0., 0., 2., 2., 1., 1., 2., 0., 2., 0., 2., 1., 0., 2., 1., 2.,\n",
       "        1., 1., 0., 2., 1., 1., 2., 0., 0., 1., 1., 0., 1., 2., 0., 2., 2., 0.,\n",
       "        0., 0., 1., 0., 0., 2., 1., 0., 2., 1., 2., 1., 0., 1., 2., 2., 0., 1.,\n",
       "        2., 2., 0., 1., 0., 0., 0., 1., 2., 2., 2., 1., 1., 0., 2., 2., 0., 1.,\n",
       "        2., 0., 2., 1., 1., 2., 0., 2., 0., 2., 2., 0., 2., 1., 1., 2., 0., 0.,\n",
       "        1., 0., 0., 1., 2., 0., 2., 2., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1.,\n",
       "        2., 2., 0., 0., 1., 2., 2., 0., 0., 0., 2., 0., 1., 0., 0., 1., 2., 1.,\n",
       "        0., 1., 2., 0., 2., 2., 1., 1., 2., 2., 2., 2., 2., 2., 2., 1., 1., 0.,\n",
       "        2., 1., 0., 1., 1., 1., 0., 0., 0., 2., 2., 1., 1., 0., 0., 1., 0., 0.,\n",
       "        2., 1., 2., 1., 0., 1., 0., 0., 1., 1., 1., 0., 1., 0., 1., 1., 2., 0.,\n",
       "        0., 0., 2., 1., 0., 1., 0., 0., 2., 1., 0., 2., 0., 0., 0., 2., 2., 2.,\n",
       "        0., 1., 2., 2., 1., 1., 1., 1., 1., 2., 0., 1., 0., 2., 0., 1., 2., 0.,\n",
       "        1., 2., 0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 2., 1., 2., 1., 2., 2.,\n",
       "        2., 1., 1., 2., 1., 2., 1., 1., 2., 1., 1., 2., 1., 0., 0., 2., 0., 0.,\n",
       "        2., 0., 0., 1., 1., 1., 1., 0., 0., 2., 0., 2., 0., 2., 2., 1., 0., 0.,\n",
       "        2., 1., 2., 2., 0., 1., 0., 2., 2., 1., 0., 0., 1., 0., 0., 2., 2., 0.,\n",
       "        1., 2., 0., 2., 0., 0., 2., 1., 1., 2., 0., 0., 1., 0., 2., 0., 2., 1.,\n",
       "        0., 2., 1., 0., 2., 1., 2., 1., 0., 2., 0., 0., 2., 2.])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.squeeze(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch 矩阵乘法运算\n",
    "\n",
    "### 1. `torch.mv`\n",
    "`torch.mv` 用于计算一个二维矩阵与一个一维向量之间的乘法。它返回一个一维向量，表示矩阵与向量的乘积。\n",
    "\n",
    "### 2. `torch.mm(A, B)`\n",
    "\n",
    "`torch.mm` 是用于两个二维矩阵之间的乘法。它要求两个输入张量都是二维的，并且第一个张量的列数必须等于第二个张量的行数。\n",
    "\n",
    "### 3. `torch.matmul(A, B)`\n",
    "\n",
    "`torch.matmul` 是一个通用的矩阵乘法函数，可以处理多种情况下的乘法运算。它可以处理任意维度的张量，并且支持广播。\n",
    "\n",
    "### 4. `torch.bmm(A, B)`\n",
    "\n",
    "`torch.bmm` 代表 batch matrix multiplication，用于批量矩阵乘法。它要求输入张量至少是三维的，并且第三维度是矩阵的维度。`bmm` 会对输入张量的第一个维度进行批量处理。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 3.0223,  3.9720, -1.8390])\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn(3, 4)  \n",
    "v = torch.randn(4)     \n",
    "w = torch.mv(A, v)\n",
    "print(w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.1880, -0.6561, -1.5751,  0.2108],\n",
      "        [ 0.3523, -1.2360,  1.5743,  0.2807],\n",
      "        [ 1.2113,  0.1812, -0.1775, -2.0566]])\n",
      "tensor([[-0.9722, -0.6501],\n",
      "        [-0.1712, -0.0815],\n",
      "        [ 0.2665,  0.4672],\n",
      "        [-0.8539,  0.3143]])\n",
      "tensor([[-1.6425, -1.3884],\n",
      "        [ 0.0491,  0.6954],\n",
      "        [ 0.5002, -1.5315]])\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn(3, 4)\n",
    "B = torch.randn(4, 2)\n",
    "print(A)\n",
    "print(B)\n",
    "C = torch.mm(A, B)\n",
    "print(C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.9237, -0.2760],\n",
      "        [ 0.9740,  0.5352],\n",
      "        [-0.9706,  2.0148]])\n",
      "tensor([[[-0.5651,  0.4980],\n",
      "         [ 1.2478,  0.7032],\n",
      "         [-1.2116, -0.7601]],\n",
      "\n",
      "        [[ 1.0463,  1.6109],\n",
      "         [ 2.3260, -0.1103],\n",
      "         [ 1.7420, -2.7521]]])\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn(3, 4)\n",
    "B = torch.randn(4, 2)\n",
    "C = torch.matmul(A, B)\n",
    "print(C)\n",
    "# 三维张量与二维张量相乘\n",
    "A = torch.randn(2, 3, 4)\n",
    "B = torch.randn(4, 2)\n",
    "C = torch.matmul(A, B)\n",
    "print(C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([10, 3, 2])\n",
      "tensor([[[-0.2231,  0.0683],\n",
      "         [-1.2908, -0.9820],\n",
      "         [ 2.2060,  1.0636]],\n",
      "\n",
      "        [[-0.3593,  3.2191],\n",
      "         [ 1.4395, -0.6196],\n",
      "         [-1.7015,  3.2343]],\n",
      "\n",
      "        [[ 1.2474, -2.7916],\n",
      "         [ 0.5665,  0.1046],\n",
      "         [-0.7204, -0.9899]],\n",
      "\n",
      "        [[-0.7077,  0.0126],\n",
      "         [ 0.4887,  1.5367],\n",
      "         [-0.2416,  0.9642]],\n",
      "\n",
      "        [[ 2.7211, -1.0547],\n",
      "         [-1.4047, -2.7027],\n",
      "         [ 0.8932, -2.7055]],\n",
      "\n",
      "        [[-0.3736,  0.9176],\n",
      "         [-0.9204, -3.9566],\n",
      "         [-2.6899,  8.6097]],\n",
      "\n",
      "        [[ 1.9016, -0.3787],\n",
      "         [-1.7463, -0.4884],\n",
      "         [-3.5153, -1.8062]],\n",
      "\n",
      "        [[ 1.0823,  1.2633],\n",
      "         [-0.5543,  1.1416],\n",
      "         [-1.4252, -1.1803]],\n",
      "\n",
      "        [[ 0.0366,  0.2722],\n",
      "         [ 2.2542,  3.4856],\n",
      "         [-1.1687, -0.4482]],\n",
      "\n",
      "        [[ 3.3335, -1.6299],\n",
      "         [ 0.7488,  1.5366],\n",
      "         [ 0.9774, -0.8057]]])\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn(10, 3, 4)\n",
    "B = torch.randn(10, 4, 2)\n",
    "C = torch.bmm(A, B)\n",
    "print(C.shape)\n",
    "print(C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "import math\n",
    "from torch.autograd import Variable\n",
    "import copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#词嵌入\n",
    "class Embeddings(nn.Module):\n",
    "    def __init__(self, d_model, vocab):\n",
    "    # d_model:词嵌入维度\n",
    "    # vocab:字典大小\n",
    "        super(Embeddings, self).__init__()\n",
    "        self.lut = nn.Embedding(vocab, d_model)\n",
    "        self.d_model = d_model\n",
    "    def forward(self, x):\n",
    "        return self.lut(x) * math.sqrt(self.d_model)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 4, 512])\n"
     ]
    }
   ],
   "source": [
    "d_model = 512  # embedding_size\n",
    "vocab = 1000  # 词典大小\n",
    "x=torch.tensor([[100, 2, 421, 508], [491, 998, 1, 221]], dtype=torch.long)\n",
    "emb = Embeddings(d_model, vocab)\n",
    "embr = emb(x)\n",
    "print(embr.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 位置编码\n",
    "\n",
    "以前的方法-表格型\n",
    "\n",
    "* 方法一：使用[0,1]范围分配\n",
    "\n",
    "这个方法的分配方式是，将0-1这个范围的，将第一个token分配0，最后一个token分配去1，其余的token按照文章的长度平均分配。具体形式如下：\n",
    "\n",
    "我喜欢吃洋葱 【0 0.16 0.32.....1】\n",
    "\n",
    "我真的不喜欢吃洋葱【0 0.125 0.25.....1】\n",
    "\n",
    "问题：我们可以看到，如果句子长度不同，那么位置编码是不一样，所以无法表示句子之间有什么相似性。\n",
    "\n",
    "* 方法二：1-n正整数范围分配\n",
    "\n",
    "这个方法比较直观，就是按照输入的顺序，一次分配给token所在的索引位置。具体形式如下：\n",
    "\n",
    "我喜欢吃洋葱 【1，2，3，4，5，6】\n",
    "\n",
    "我真的不喜欢吃洋葱【1，2，3，4，5，6，7】\n",
    "\n",
    "问题：往往句子越长，后面的值越大，数字越大说明这个位置占的权重也越大，这样的方式无法凸显每个位置的真实的权重。\n",
    "\n",
    "https://erdem.pl/2021/05/understanding-positional-encoding-in-transformers\n",
    "\n",
    "https://www.cnblogs.com/yanshw/p/16740972.html\n",
    "\n",
    "https://www.cnblogs.com/ghj1976/p/li-jietransformer-de-wei-zhi-bian-ma.html\n",
    "\n",
    "https://blog.csdn.net/Kaiyuan_sjtu/article/details/119621613"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"31.jpg\" width=\"800\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"31.jpg\",width=800)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"32.jpg\" width=\"800\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"32.jpg\",width=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformer算法中的位置编码与其他方法的比较（chatgpt）\n",
    "\n",
    "Transformer算法中的位置编码(Positional Encoding)与RNN、LSTM处理顺序信息的方法具有以下优缺点:\n",
    "\n",
    "**位置编码的优点:**\n",
    "\n",
    "- 计算效率高,可以完全并行\n",
    "- 为模型带来一定位置不变性,对小的位置变化更鲁棒\n",
    "- 使用周期函数,使模型更好地学习位置信息\n",
    "- 避免位置信息过于绝对化\n",
    "- 较RNN和LSTM更简单,只编码位置信息\n",
    "\n",
    "**位置编码的缺点:**\n",
    "\n",
    "- 需要设置最大序列长度,长度过长会导致计算复杂度提升\n",
    "- 仅编码相对位置,无法表示绝对位置\n",
    "- 对语序变化不如RNN等顺序敏感\n",
    "- 需搭配注意力机制,单独使用效果较弱\n",
    "\n",
    "**RNN和LSTM的优点:** \n",
    "\n",
    "- 可以表示绝对位置信息\n",
    "- 对语序变化更敏感\n",
    "- LSTM可以捕捉长程依赖关系\n",
    "\n",
    "**RNN和LSTM的缺点:**\n",
    "\n",
    "- 计算复杂度高,无法并行计算\n",
    "- 对位置信息过于敏感\n",
    "- 隐状态编码了所有历史,计算资源利用率低\n",
    "\n",
    "总之,位置编码更高效并具有一定位置鲁棒性,但RNN和LSTM可以建模绝对位置并对语序更敏感。两者互补的特点可根据任务需求进行选择。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#np.zeros(shape=(60,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.00000000e+00  1.00000000e+00  0.00000000e+00 ...  1.00000000e+00\n",
      "   0.00000000e+00  1.00000000e+00]\n",
      " [ 8.41470985e-01  5.40302306e-01  7.61720408e-01 ...  9.99999991e-01\n",
      "   1.15478198e-04  9.99999993e-01]\n",
      " [ 9.09297427e-01 -4.16146837e-01  9.87046251e-01 ...  9.99999964e-01\n",
      "   2.30956395e-04  9.99999973e-01]\n",
      " ...\n",
      " [ 1.23573123e-01 -9.92335469e-01  1.39920673e-01 ...  9.99980359e-01\n",
      "   5.42744868e-03  9.99985271e-01]\n",
      " [-7.68254661e-01 -6.40144339e-01 -6.63571724e-01 ...  9.99979514e-01\n",
      "   5.54292514e-03  9.99984638e-01]\n",
      " [-9.53752653e-01  3.00592544e-01 -9.99784705e-01 ...  9.99978652e-01\n",
      "   5.65840153e-03  9.99983991e-01]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA58AAAGNCAYAAACWkMXFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB0cUlEQVR4nO3deXhU5d3/8e8syWQhCQTMJgGChkUQZVG2qlg0ilutWrFaqi1iedAqpT5aavuY+liotkVc6kJri1YFnv6QSltEcUMtoMjighZRgwRICFv2ZGYyc35/UFLH+d44ZxaYJO/Xdc116Sd3ztxzziThnnPmMw7LsiwBAAAAACCBnMd6AgAAAACAzo/FJwAAAAAg4Vh8AgAAAAASjsUnAAAAACDhWHwCAAAAABKOxScAAAAAIOFYfAIAAAAAEo7FJwAAAAAg4Vh8AgAAAAASjsUnAAAAACDhWHwCAAAAQBJ7/fXX5eKLL5aioiJxOBzy17/+9Su/Z/Xq1TJy5EhJS0uT/v37y6OPPho2ZunSpXLSSSeJx+ORk046SZYtW5aA2f8Hi08AAAAASGJNTU1yyimnyEMPPRTR+IqKCrngggvkjDPOkE2bNslPf/pTufnmm2Xp0qXtY9auXSuTJ0+WKVOmyLvvvitTpkyRK6+8Ut56661EPQxxWJZlJWzrAAAAAIC4cTgcsmzZMrn00kuNY26//XZZvny5fPTRR+3Z9OnT5d1335W1a9eKiMjkyZOlvr5enn/++fYx559/vvTo0UMWLVqUkLm7E7JVAAAAAOhkWltbxefzxWVblmWJw+EIyTwej3g8npi3vXbtWikrKwvJzjvvPHn88cfF7/dLSkqKrF27Vn70ox+FjZk/f37M92/C4hMAAAAAvkJra6uU9O0m1TWBuGyvW7du0tjYGJLdeeedUl5eHvO2q6urJT8/PyTLz8+XtrY22bdvnxQWFhrHVFdXx3z/Jiw+AQAAAOAr+Hw+qa4JSMWGvpKdFVt1Tn1DUEpGfi6VlZWSnZ3dnsfjrOdhXz6revjdll/MtTFfzuKJxScAAAAARCg7yxnz4rN9W9nZIYvPeCkoKAg7g1lTUyNut1t69ux5xDFfPhsaT7TdAgAAAECEAlYwLrdEGjt2rKxatSoke/HFF2XUqFGSkpJyxDHjxo1L2Lw48wkAAAAAEQqKJUGJ7QND7H5/Y2OjfPLJJ+3/X1FRIZs3b5bc3Fzp06ePzJ49W3bt2iVPPvmkiBxqtn3ooYdk1qxZMm3aNFm7dq08/vjjIS22t9xyi5x55plyzz33yDe+8Q157rnn5KWXXpI333wzpsd2JJz5BAAAAIAk9s4778jw4cNl+PDhIiIya9YsGT58uPzP//yPiIhUVVXJjh072seXlJTIihUr5LXXXpNTTz1V/vd//1ceeOABufzyy9vHjBs3ThYvXix/+tOfZNiwYbJw4UJZsmSJjB49OmGPg8/5BAAAAICvUF9fLzk5ObJ7a++4FA4VDdwpdXV1CXnPZ7LislsAAAAAiFDAsiQQ4/m7WL+/o+KyWwAAAABAwnHmEwAAAAAidCwKhzoLFp8AAAAAEKGgWBJg8RkVLrsFAAAAACQcZz4BAAAAIEJcdhs9Fp8AAAAAECHabqPHZbcAAAAAgIRL6sXnww8/LCUlJZKWliYjR46UN95441hPCTbNnTtXTjvtNMnKypK8vDy59NJLZevWrSFjLMuS8vJyKSoqkvT0dJkwYYJs2bLlGM0Y0Zg7d644HA6ZOXNme8Zx7dh27dol3/nOd6Rnz56SkZEhp556qmzYsKH96xzfjqmtrU1+9rOfSUlJiaSnp0v//v3lrrvukmAw2D6GY9sxvP7663LxxRdLUVGROBwO+etf/xry9UiOo9frlR/+8IfSq1cvyczMlEsuuUR27tx5FB8FNEc6tn6/X26//XY5+eSTJTMzU4qKiuS73/2u7N69O2QbHNvECsbp1hUl7eJzyZIlMnPmTLnjjjtk06ZNcsYZZ8ikSZNkx44dx3pqsGH16tVy4403yrp162TVqlXS1tYmZWVl0tTU1D7m3nvvlXnz5slDDz0k69evl4KCAjn33HOloaHhGM4ckVq/fr0sWLBAhg0bFpJzXDuugwcPyvjx4yUlJUWef/55+fDDD+W3v/2tdO/evX0Mx7djuueee+TRRx+Vhx56SD766CO599575de//rU8+OCD7WM4th1DU1OTnHLKKfLQQw+pX4/kOM6cOVOWLVsmixcvljfffFMaGxvloosukkAgcLQeBhRHOrbNzc2yceNG+fnPfy4bN26UZ599Vj7++GO55JJLQsZxbBMr8O+221hvXZKVpE4//XRr+vTpIdmgQYOsn/zkJ8doRoiHmpoaS0Ss1atXW5ZlWcFg0CooKLB+9atftY9pbW21cnJyrEcfffRYTRMRamhosEpLS61Vq1ZZZ511lnXLLbdYlsVx7ehuv/1262tf+5rx6xzfjuvCCy+0vv/974dkl112mfWd73zHsiyObUclItayZcva/z+S41hbW2ulpKRYixcvbh+za9cuy+l0WitXrjxqc8eRffnYat5++21LRKzPP//csiyObSLV1dVZImK992GeVVFZENPtvQ/zLBGx6urqjvXDOqqS8synz+eTDRs2SFlZWUheVlYma9asOUazQjzU1dWJiEhubq6IiFRUVEh1dXXIsfZ4PHLWWWdxrDuAG2+8US688EI555xzQnKOa8e2fPlyGTVqlHzrW9+SvLw8GT58uPz+979v/zrHt+P62te+Ji+//LJ8/PHHIiLy7rvvyptvvikXXHCBiHBsO4tIjuOGDRvE7/eHjCkqKpKhQ4dyrDuYuro6cTgc7VencGyRzJKy7Xbfvn0SCAQkPz8/JM/Pz5fq6upjNCvEyrIsmTVrlnzta1+ToUOHioi0H0/tWH/++edHfY6I3OLFi2Xjxo2yfv36sK9xXDu2zz77TB555BGZNWuW/PSnP5W3335bbr75ZvF4PPLd736X49uB3X777VJXVyeDBg0Sl8slgUBAfvnLX8q3v/1tEeFnt7OI5DhWV1dLamqq9OjRI2wM/9bqOFpbW+UnP/mJXH311ZKdnS0iHNujIR7v2eyq7/lMysXnYQ6HI+T/LcsKy9Bx3HTTTfLee+/Jm2++GfY1jnXHUllZKbfccou8+OKLkpaWZhzHce2YgsGgjBo1SubMmSMiIsOHD5ctW7bII488It/97nfbx3F8O54lS5bIU089Jc8884wMGTJENm/eLDNnzpSioiK59tpr28dxbDuHaI4jx7rj8Pv9ctVVV0kwGJSHH374K8dzbOMnKA4JSGz7Mhjj93dUSXnZba9evcTlcoW9OlNTUxP2Kh46hh/+8IeyfPlyefXVV6V3797teUFBgYgIx7qD2bBhg9TU1MjIkSPF7XaL2+2W1atXywMPPCBut7v92HFcO6bCwkI56aSTQrLBgwe3F77xc9tx/fd//7f85Cc/kauuukpOPvlkmTJlivzoRz+SuXPnigjHtrOI5DgWFBSIz+eTgwcPGscgefn9frnyyiuloqJCVq1a1X7WU4Rji+SWlIvP1NRUGTlypKxatSokX7VqlYwbN+4YzQrRsCxLbrrpJnn22WfllVdekZKSkpCvl5SUSEFBQcix9vl8snr1ao51Eps4caK8//77snnz5vbbqFGj5JprrpHNmzdL//79Oa4d2Pjx48M+Eunjjz+Wvn37igg/tx1Zc3OzOJ2hf/pdLlf7R61wbDuHSI7jyJEjJSUlJWRMVVWVfPDBBxzrJHd44blt2zZ56aWXpGfPniFf59gmXtCKz60rStrLbmfNmiVTpkyRUaNGydixY2XBggWyY8cOmT59+rGeGmy48cYb5ZlnnpHnnntOsrKy2l+FzcnJkfT09PbPhpwzZ46UlpZKaWmpzJkzRzIyMuTqq68+xrOHSVZWVvv7dg/LzMyUnj17tucc147rRz/6kYwbN07mzJkjV155pbz99tuyYMECWbBggYgIP7cd2MUXXyy//OUvpU+fPjJkyBDZtGmTzJs3T77//e+LCMe2I2lsbJRPPvmk/f8rKipk8+bNkpubK3369PnK45iTkyNTp06VH//4x9KzZ0/Jzc2VW2+9VU4++eSwEjkcXUc6tkVFRXLFFVfIxo0b5e9//7sEAoH2f1vl5uZKamoqx/YoCMThsttYv7/DOlY1u5H43e9+Z/Xt29dKTU21RowY0f7xHOg4RES9/elPf2ofEwwGrTvvvNMqKCiwPB6PdeaZZ1rvv//+sZs0ovLFj1qxLI5rR/e3v/3NGjp0qOXxeKxBgwZZCxYsCPk6x7djqq+vt2655RarT58+VlpamtW/f3/rjjvusLxeb/sYjm3H8Oqrr6p/X6+99lrLsiI7ji0tLdZNN91k5ebmWunp6dZFF11k7dix4xg8GnzRkY5tRUWF8d9Wr776avs2OLaJcfijVt7aUmBt2VEU0+2tLQVd8qNWHJZlddGTvgAAAAAQmfr6esnJyZE1WwqlW1Zs715sbAjKuCFVUldXF/Ke3c4uaS+7BQAAAIBkE7QcErRibLuN8fs7qqQsHAIAAAAAdC6c+QQAAACACFE4FD0WnwAAAAAQoYA4JRDjBaSBOM2lo+GyWwAAAABAwnHmEwAAAAAiZMWhcMiicCj5eL1eKS8vF6/Xe6yngjjj2HZuHN/Oi2PbeXFsOy+ObefFsT02Dr/nM9ZbV5TUn/N5+LN0utrn33QFHNvOjePbeXFsOy+ObefFse28OLZH1+H9/fx7JZIZ4+d8NjUEZdKwii537JL6zCcAAAAAoHPgPZ8AAAAAEKGgOCQY4zm8oCTtxacJlbDF58MPPyy//vWvpaqqSoYMGSLz58+XM8444yu/LxgMyu7duyUrK0saGhpE5NApbnQuh48px7Zz4vh2Xhzbzotj23lxbDuvjnZsLcuShoYGKSoqEqez416Ayed8Ri8h7/lcsmSJTJkyRR5++GEZP368PPbYY/KHP/xBPvzwQ+nTp88Rv3fnzp1SXFwc7ykBAAAASAKVlZXSu3fvYz0N2w6/53P5eydIZpYrpm01NQTkkmGfdrn3fCZk8Tl69GgZMWKEPPLII+3Z4MGD5dJLL5W5c+ce8Xvr6uqke/fu0vvOn4kzLS3ka+9e/kf1e05Z+n0178jjk2kujD8645NpLow/tuOTaS6MPzrjk2kujD8645NpLow/tuOTaS6JHl/fGJS+I7ZLbW2t5OTkqN+XzA4vPpe9WxqXxec3T9nW5Rafcb/s1ufzyYYNG+QnP/lJSF5WViZr1qwJG+/1ekPqoQ9fautMSwtbfGYbWqW+PK4zjE+muTD+6IxPprkw/tiOT6a5MP7ojE+muTD+6IxPprkw/tiOT6a5HI3xIiIOR8e+5PTQez5jewyxfn9HFfeLrfft2yeBQEDy8/ND8vz8fKmurg4bP3fuXMnJyWm/ccktAAAAAHQ+cV98HvblVzQsy1Jf5Zg9e7bU1dW13yorKxM1JQAAAACISVCcEojxFmtbbkcV98tue/XqJS6XK+wsZ01NTdjZUBERj8cjHo8n3tMAAAAAgLgLWE4JWLEtHgPxr93pEOK++ExNTZWRI0fKqlWr5Jvf/GZ7vmrVKvnGN74R8XYWXvSodPvSteI37dI/quW+i59U818fOEHN//v8v6n54oYeav6diW+o+cst+huNzztjs5pv/sJ7W79o5OnbwrJP/Y3q2NJT9TPDO9v08YUn1aj5vkCTmncvPaDmdcEWNc8oqVfz5qBPzVOL9fv1Wn41dxXq9+u3Amouefo+No23eurzDFhBNQ/20OdpHJ/dpubq2G6Gx2Qan2lzfIY+R+P4dJvj0+yNtzz2fulaqTbHpyR4vDtx4y2bHQZ2x9t+sbWjj7f7tppkGp9Mc2F8fMcn01wYf2zHJ9NcjsZ4RM3OR1led9118sQTT4TlJ510kmzZskVERBYuXCjf+973wsa0tLRImuG9vLFKyOd8zpo1S6ZMmSKjRo2SsWPHyoIFC2THjh0yffr0RNwdAAAAABwVwThcNhsUey9iL1myRGbOnBnyUZaTJk0yfpTl/fffL7/61a/a/7+trU1OOeUU+da3vhUyLjs7W7Zu3RqSJWrhKZKgxefkyZNl//79ctddd0lVVZUMHTpUVqxYIX379k3E3QEAAADAURGwHBKwYjvla/f7582bJ1OnTpXrr79eRETmz58vL7zwgjzyyCPqR1keLnM97K9//ascPHgw7Eynw+GQgoKCKB5BdBL2TtcZM2bI9u3bxev1yoYNG+TMM89M1F0BAAAAQIdTX18fcvMqb9M7/FGWZWVlIbnpoyw1jz/+uJxzzjlhJwMbGxulb9++0rt3b7noootk06ZN0T+YCHTNmiUAAAAAiEKsTbeHbyIixcXFIR87qZ3FtPtRll9WVVUlzz//fPtZ08MGDRokCxculOXLl8uiRYskLS1Nxo8fL9u2hffRxEtCLrsFAAAAgM4oaDklGGPbbfDfbbeVlZWSnZ3dnh/pU0Ai/SjLL1u4cKF0795dLr300pB8zJgxMmbMmPb/Hz9+vIwYMUIefPBBeeCBByJ5GLYl7eIzz+WTLFfoQX1n/nB17P33rlXznyw8T80/vPFhNS9ZfoOab7noITUfvf46Nf/HyAVqflPFFWr+0+J/hGUP7pugjp3WW2/eXdYwRM0nF29Q89UthWp+Xu9/qfm7vnQ1H1P0uZp/0qY3nw4pqFLznW16S21J3n413xfQW3ALetWpeV2wVc2799DbdxstfT6ZOfp2vJbeauvJ1rejtfu6u+lNuqamXmemfp+m5l1Hur3x4tFz4/hUm223KTbH222XTaL2WhERsdNI67K5bafNuSd6vM23wSR6fNI1PSayBRMAkHBfPHMZ/TYO/W3Nzs4OWXxq7H6U5RdZliV//OMfZcqUKZKamnrEsU6nU0477bSEnvnkslsAAAAASFJf/CjLL1q1apWMGzfuiN+7evVq+eSTT2Tq1KlfeT+WZcnmzZulsFA/SRUPSXvmEwAAAACSTVDst9Vq27Djqz7Kcvbs2bJr1y558sknQ77v8ccfl9GjR8vQoUPDtvmLX/xCxowZI6WlpVJfXy8PPPCAbN68WX73u99F+7C+EotPAAAAAIhQfD7n0973f9VHWVZVVcmOHTtCvqeurk6WLl0q999/v7rN2tpaueGGG6S6ulpycnJk+PDh8vrrr8vpp58e3YOKAItPAAAAAEhyM2bMkBkzZqhfW7hwYViWk5Mjzc3Nxu3dd999ct9998VrehFh8QkAAAAAEQpYTgnE2HYb6/d3VEm7+Jz0+n+JMz0tJCt95i117PSbz1Dzkj9+puYrv6dXGJ/4tN422nihnqctz1HzPqO7qfnHr/VX8zHTwmswv73pVHXs3ee/qeZnbpyk5n855XE1v33HpWr+o+NfVPPldSPUfGL3D9V8TfMJaj6uh35M3vcVqPmwHrvU/LO2DDUf1L1GzfcE9B/wft0PqPmBgN4wm5/doOZ1QZ+a98jSX21qDoY/p7plmpp0Dc+/dP0+Tc27KWl63ib6Y3UZxgdFbz51evTtGNt3De24dtt0jePtNsbabpi1N9yysX277bK2r/xJ+Hib83ckdnzSte8mUjI19QJAJxUUhwRj/IUY6/d3VF1zyQ0AAAAAOKqS9swnAAAAACQbLruNHotPAAAAAIhQQJwSiPEC0li/v6Pqmo8aAAAAAHBUceYTAAAAACIUtBwSjLFtLtbv76iSdvE5YH6juF2hLZ9tY4epYzf8SW8+zT+4Sc1nvPkdNS99c6Oa37G7TM3zVlao+co79Dbd4pda1Hzf95vCsl7/TFHHdrsgTc2bN/VU8xNG6c27G7b2U/NTSvQG1R9WDlTzG07R23fj1aY7MnO7mr/fWqzmQ7rtVvNt/uPU/IRu+9R8d0B/TvXrZmjHDYY3FouIFGbWq3ltMLyhNTdTb8ZtCOqts9kZpnZcfXxGmn5s/ZbeUuvx6C27pvHuVD03tek6DeONbbop+ngTR4rNdly3fr9xa9O10wCbwCZdkSjadBPdiGr3Ghzb80lw+65Ndv690eH/bUKbLoBOKBiHy26DXfQC1KRdfAIAAABAsglaTgnGWBgU6/d3VF3zUQMAAAAAjirOfAIAAABAhALikECM7xOI9fs7KhafAAAAABAhLruNXtd81AAAAACAoyppz3wGKyol6AhtfN2zpL86tmjKZ2q+79vD1bzfM3qLp7tPbzV/4wU971u1Vs3/Z+slat5z/Udq/mTdyWHZcev0Ftb3fHrDad5GveG0Oag3nGZ/kKrmpjbd+m091LzPyHQ1f7dS32el/fTG37f39VXza0rXqfmDeyaq+eU931Hzjc391HxQepWaf+rLU/OSDP247GrLVvOidL3t9kAwfP/npzeoYxsMdZc90vR92WRoZ81K86p5q6G9NsPUdmtor7XbjptiaK8Nij5/l1vPze249sY7DNs3MoyPSzuu3SZdu+2sdtt0bbbjJl2bbsLHJ0+brt123C7XpgsAcRCQ2C+btdfh33kk7eITAAAAAJINl91Gr2s+agAAAADAUcWZTwAAAACIUMBySiDGM5exfn9HxeITAAAAACJkiUOCMb7n0+qib1rvmktuAAAAAMBRlbRnPvdcd6q4PKHNq/8cNU8de6XnQjUvmfqxmtedWavmO348Ws37/U1vIXUMG6Tmra/kqrnVVqHmj37wtbCs/7YP1bF/PjBWzbPerVbzTT79EPf8QG8+rQvqDao5H+uvzqQ4XGru2K634PY8S88/39VLzYsH6c2h7+8vVPP/Ljig5k82jFPzcQXb1PzVxsFqfoJnj5pX+nuqeXGaPp/qQHg7bmFanTr2QEBvIO6V1qjmDUH9NaXuhnbcZktv3uzm0Z8jXkOba0aqqe1WH5+aojc0G9txU+2Nd7rstek6bbbpOmw20prGq+24dtti7b6MaLud1e727Q23kmw+tiVy+wlsxo1Gl2vH7eiPF0BCcNlt9JJ28QkAAAAAySZoOSQY46txsX5/R9U1l9wAAAAAgKOKM58AAAAAEKGAOCUQ4zm8WL+/o2LxCQAAAAAR4rLb6LH4BAAAAIAIBcUpwRjPXMb6/R1V0i4+r5n6oqR1C53eSy16I2rldQPV/J2S+9X8svyL1fz0y99T852/adLvd7bePNtnRa2aW6fo80xf2y08dOhPyGX/GqrmJ+x4X82XHNAbfNO36q2tm7yZat59m0/NDwaa1TxbL/YVl+FxpexM1bfj1Jte9+zJUfP8Yfr2tx08Ts2LeutNxlsb89X87G4fqflLDUPUfHD6bjXf5Q9vRC7w6G23NYEsNT8uVW+7rQt61LynR38eNwT1xuKc1FY1bzW042am6s8Rv6EtNt1uO67bXnttSoo+PmCYv8tlaruNVzuuPl4fa6MZV4R23DhvP+nG29o27bjHVFd7vABgU9IuPgEAAAAg2QQshwRifHUt1u/vqFh8AgAAAECEeM9n9LrmxcYAAAAAgKOKM58AAAAAECHLckrQiu0cnhXj93dULD4BAAAAIEIBcUggxoaxWL+/o0raxecPcrZLdlboKwInP3qTOvay77yh5mtb9dbPXVf0V/Mlveep+ZU9L1TzIRdsVfO6X9Wq+e4f682zx78W3rjqGKzPMWNDhpo7nPoTeOUng9W8f9WHav6PulPUPO2zvWr+oV9vo82u0JtP64Itat6tUo3N7bhVejtuN4d+zPfvMzTGGvbb9rrwNloRczvuZ016E/M5WVvU/NXG8ONygkdvIN7blq3mean6XPYH9cbi3BRT262haThVP1ZNQf2YZKbox9zUjpueYmi7NbTFelLa9PGGBli3zfZat6FN19SO63Ta276pwVZtx7XZXks77pElvB03kRI9F9pxj52u9FgB4N+SdvEJAAAAAMkmaMVeGBRMrtf+jhoWnwAAAAAQoWAc3vMZ6/d3VF3zUQMAAAAAjioWnwAAAAAQoaA44nKz6+GHH5aSkhJJS0uTkSNHyhtv6L03IiKvvfaaOByOsNu//vWvkHFLly6Vk046STwej5x00kmybNky2/Oyg8UnAAAAAEQoYDnicrNjyZIlMnPmTLnjjjtk06ZNcsYZZ8ikSZNkx44dR/y+rVu3SlVVVfuttLS0/Wtr166VyZMny5QpU+Tdd9+VKVOmyJVXXilvvfVWVPslEkn7ns9vbTtf3JmhzaX9HtSbQ+/+r/fVvOSvN6j58G9tU/O9Ab1N82BZqZo/2uc3av6DtDI17/Z1vc3U8eCusKzm2uHq2LyNrWru7Fes5u4t3dTcCujNni9VDlTz/KoKNV+ttLaKiKTtqFXzCr/+ekfWTn3fNwf1BtWMKv0H1tSO69xraMd16u24+w7q7bi5hpdrdjZ2V/PjXHrD7I6W8Dbd8Zkfq2Pfbj5BzQtTatV8f5t+zHukNKt5bVBvUO6eorfdNlgpap6doj8349WOm+Y2tN3abMcNGMa7bLfjGppkDUztuOpYO824IlG019KOeyQJb8dNZMsp7bgA0CnNmzdPpk6dKtdff72IiMyfP19eeOEFeeSRR2Tu3LnG78vLy5Pu3burX5s/f76ce+65Mnv2bBERmT17tqxevVrmz58vixYtivtjEOHMJwAAAABE7HDhUKw3EZH6+vqQm9frDbs/n88nGzZskLKy0BNcZWVlsmbNmiPOdfjw4VJYWCgTJ06UV199NeRra9euDdvmeeed95XbjAWLTwAAAACIUFAcErRivP37UpXi4mLJyclpv2lnMfft2yeBQEDy8/ND8vz8fKmurlbnWFhYKAsWLJClS5fKs88+KwMHDpSJEyfK66+/3j6murra1jbjIWkvuwUAAACAZGNFWRj05W2IiFRWVkp2dnZ77vHobwkTEXE4Qu/Tsqyw7LCBAwfKwIH/eTvd2LFjpbKyUn7zm9/ImWeeGdU244EznwAAAABwDGRnZ4fctMVnr169xOVyhZ2RrKmpCTtzeSRjxoyRbdv+031TUFAQ8zbtsr34fP311+Xiiy+WoqIicTgc8te//jXk65ZlSXl5uRQVFUl6erpMmDBBtmzRi4IAAAAAoCOJ+ZLbf98ilZqaKiNHjpRVq1aF5KtWrZJx48ZFvJ1NmzZJYWFh+/+PHTs2bJsvvviirW3aZfuy26amJjnllFPke9/7nlx++eVhX7/33ntl3rx5snDhQhkwYIDcfffdcu6558rWrVslK0tvENW0PlAo7pS0kCyz22517IK6IjUf9FiDmj/+9+fU/KIt31Xz/RfrrZ+FrnQ1958+SM3vHvCEmt/rHRaWHRwf/mZjEZH8ZyvVvP6s/mrec4veauvKO07NG7Z1V/M85c3PIiKv7h2g5inVe9V8Q2tfNU/fpbfCVgX0RtTMav1x+S09T9+r/4CnOFxqHtyvt+NmOfV8b73eMJvr1OezuyknLOuZr++Dam/4WBGRYel6tfZWb6Ga93LrPw+1Ab3t1tSO2xBMU3NT263X0vdxhls/tn5DmWaa26/mPkM7bqrL9BzRG11TDOPttuMGDPNxGRtsw7djpxlXRMRhaKM1teOaxhvZbcdNeFtscrXRJrQdN+HttYnefnK143ZoNPsCSeOLhUGxbMOOWbNmyZQpU2TUqFEyduxYWbBggezYsUOmT58uIoeaanft2iVPPvmkiBxqsu3Xr58MGTJEfD6fPPXUU7J06VJZunRp+zZvueUWOfPMM+Wee+6Rb3zjG/Lcc8/JSy+9JG+++WZMj+1IbC8+J02aJJMmTVK/ZlmWzJ8/X+644w657LLLRETkiSeekPz8fHnmmWfkBz/4QWyzBQAAAIAuZvLkybJ//3656667pKqqSoYOHSorVqyQvn0PndipqqoK+cxPn88nt956q+zatUvS09NlyJAh8o9//EMuuOCC9jHjxo2TxYsXy89+9jP5+c9/LieccIIsWbJERo8enbDHEdfCoYqKCqmurg6p7PV4PHLWWWfJmjVr1MWn1+sNqRSur6+P55QAAAAAIG7sXjZr2oZdM2bMkBkzZqhfW7hwYcj/33bbbXLbbbd95TavuOIKueKKK2zPJVpxLRw6/IZVO5W9c+fODakXLi4ujueUAAAAACBugv9uu4311hUlpO3WTmXv7Nmzpa6urv1WWam/pxEAAAAA0HHF9bLbgoICETl0BvSLTUpHquz1eDxH/DwbAAAAAEgWx+qy284grovPkpISKSgokFWrVsnw4cNF5NCbXVevXi333HOPrW15XtwobkdKSPav+8aoYz/5f33UvN+7a9Xcb2iA9C/SF8i/+ZneUvvgwVI1r5yoL6YnputtmvNKwhtgrx62Xh27fq/eHFoz4kQ1P/EJvXW27US9ITjnY/0HwZmhN6J+/Lm+zwY07FTzNXX6PJ3V+9X8X/5eap5RpbfvHgzqjavpNfYaFz0H9P3s+dJz8rDWWr0BNsup/4jVNIa34+Y69fbX6la9Jbqnq1HN9/n18X0z96n53rZsNc9x6S3PDUG95TnLre/7JkvfZ1lu/Ri2xqkdNz1Fb8f1m9px3frPZ9Aw3m1opNXaa0XsteOa22v1bdhtr7U93u7fR8M1NcZ2XLvtux29ATaROvLcj4Iu9W+9rvRYgaOMxWf0bC8+Gxsb5ZNPPmn//4qKCtm8ebPk5uZKnz59ZObMmTJnzhwpLS2V0tJSmTNnjmRkZMjVV18d14kDAAAAADoO24vPd955R84+++z2/581a5aIiFx77bWycOFCue2226SlpUVmzJghBw8elNGjR8uLL75o6zM+AQAAACAZceYzerYXnxMmTBDLcBmayKGyofLycikvL49lXgAAAACQdFh8Ri8hbbcAAAAAAHxRXAuHAAAAAKAzs0Ri/pxOm1V7nUbSLj69542QQEpog+jSS+9Xx94x4Qo1bz5vlJp/d9vxat7ruQ/V/JK5zWpesuocNR9+1idqXuHX20kPjCkIy27IfUoduyGjTM17Dq9Rc+uuXWq+f8JwNe/xsd5A6ijSW23TK1LV3GTjnt5qnn+wQs3fbQ5vAhYRSalpUPPdbfpTOmNvm5o3B/UGVY9evmvkqtXv19SO29AY3hib5dR/idU06++X7u7Uj1WN1zA+S38ef9xWqOaFKbVqXh8wNPu69LZbUztupqHttskyHEO33l7baukXcHhc+jHXtyLiceltt6Z27BRDO27AMN5pox3X1IxrYtq2icNwzUvQ9KfQdhttgttrbY9P8J94m/Ox7Mwn2a7KSnjTcHL9c6yLXhUH4Ctw2W30knbxCQAAAADJhsVn9HjPJwAAAAAg4TjzCQAAAAAR4sxn9Fh8AgAAAECEWHxGL2kXnxk37xZ3pick6+XSq0ICu/eo+d77stU89f/6qHl+6yY1f9ur32+/5XoxQvkFf1PzOdXnqfme8eHb6ePupo6VAf3U+Pp+r6j5/7WGlxmJiBwcqheUFKzcq+bNg/TCoewKfTuuHH3f1+7S8zyvXj6zsa5YzWX/QTX+2J+n5p69ehnOAUPhUNoB/XEFLD1PrdV/gaQ4XPp26sOLiDIM5UQHm/XCniynXniz35up5tlOfR/s8+vPtcFpu9V8t7+Hmue4WtS8KehR824u/Zi3Wvp+SDf8/PsN7x5IM4039JmYCooChs81TjEVFBmeI25DKZBWUOQ0FPyY5mIar5UZiYg4bJa6OGwWDtkdbyqxMf28JbzQyK4EFibZKieyue2oxqPz4rkA4ChI2sUnAAAAACQby3KIFeOZy1i/v6Ni8QkAAAAAEQqKI+bP+Yz1+zsq2m4BAAAAAAnHmU8AAAAAiBCFQ9Fj8QkAAAAAEeI9n9FL2sXnkhNfkOys0KuCT3zxFnVswbf0dswVo36j5tN/8E01r7/wFDW/6UO9cbXn61vUfFhqmpq/+sbJan721z4Iy7b49ObQfcNz1Pzibp+q+dLsAWpeMrhKzQPVNWpee7G+D3ptblZzyT9OjTMqDU85p94K+6+9estu7/ptar6lpbeauw40qvnugN7EmnZAbzJtsfR2XE+tGhu568Mfr8eh75vmRn2OGQ79l9b+lgw1z3Hq7bIHfPr4LKf+HDzQpu/jvp59at4Q1H8eTG23zYZ23Ey3qR1Xf+6kGdprfZb+bgNje62aiqQY2msN/azidpm+Es5l3Laem9puTRyG7RvHG/4+BpWm3kPfYGvziW+vTfR8YJbwpuHkOlZd6t+SXemxAoibpF18AgAAAECy4bLb6LH4BAAAAIAIcdlt9Gi7BQAAAAAkHGc+AQAAACBCVhwuu+2qZz5ZfAIAAABAhCwRsWLsO0uuurSjJ2kXn4/U9pe0ttDpDb63Xh0b/J3euGq6pjhYW6fmB6/WG1E9f+9l2FClGr/t1fsxe7+st2necsVLYdn8PeeoY/eP1Fsq81yZam6VHK/mk49freZLvXlqXjdAn3vR3/arefMgvaW22079R83VTZ9/U7Xhcfn11tkt9YVqLrX6c2e7Xz+2nv2t+maCeoOqp1Y/LgFLz1Pqw1/tcjn0Z2ywSW9zznDoeX2L3i6baWg4rTW03WY79X1wsE0fPzRdH7/b30PNs1z6+CZD222GUz/mrZa+H9Jd+s+h3/CbIc003vDXwWNo0w0Y/hoZ23SV54ip7TZg+FNlars1zcU03tSm67DZKOqw3b4bn7Zb089bwtt07bKzfZtzsZKtORgAOqGgOMQR4y/EYBf9hcp7PgEAAAAACZe0Zz4BAAAAINnQdhs9Fp8AAAAAEKGg5RAHn/MZFS67BQAAAAAkHGc+AQAAACBClhWHttsuWnebtIvPxX88R1ypoa2d+Z9tUMf+vwF6c+u4d6apeW5ZNzV/csTv1Px/Zl2u5s1nDVXz8u25ap6x9mM1H5Ya3k760oYh6tjhwz5T8wq/3tRbOyRHzc/P1OeyLKOfmueX7lPz4J69al5/fm817/GxV80lr6cap1cZnqIO/VKFTw/o7bX5DRVq/klrgZq7avUG5X0BvVk1rVZvMvVaeiNqaoMa63NpcKl5ikPPW5oNbbGGfVbbmq7mWYZ22Tq/aXyLPj5QpOaFKbVq3hRMVfMMlz6fZkM7brphfKul77c0Q3utz9IvEDG216qpSIqhwVZL3S5Da6uB07htPTe13Zo4DNs3jjdcSRQ0FcvbblxNskZXu/OBWcKbhpPnWHXRK+6AToH3fEaPy24BAAAAAAmXtGc+AQAAACDZcOYzeiw+AQAAACBCtN1Gj8tuAQAAACDJPfzww1JSUiJpaWkycuRIeeONN4xjn332WTn33HPluOOOk+zsbBk7dqy88MILIWMWLlwoDocj7Nba2pqwx8DiEwAAAAAidLjtNtabHUuWLJGZM2fKHXfcIZs2bZIzzjhDJk2aJDt27FDHv/7663LuuefKihUrZMOGDXL22WfLxRdfLJs2bQoZl52dLVVVVSG3tLTwMtR4SdrLbvP+uEncjtBm0erpo9SxH/rfVPPcx/RW253f0Vsth6XqLZhtn1eq+ef/rbd4el7tp+Z96tbq82kLb6rNX6O/LnBD2etqvqhupJrvH6qf0u/j1veN43i9/fX8oo/UfE2r3kzaUKLGkv9qnZr7C7ureUa1/pPpTNcbV2v36Y8rz6u37G5rzlNzR73eHrwroLcHp9bqHaeNlp6n1oc/roClN4qmNOrH0NR2G2jSf6zTDOMbWk3tuHqba51P/4WU6dT3cV2bfqwGpFWp+d62bDXv5tJfhWu19AbidJe+7/2GtluPU/+94De8RpdqGB8w/DExteMGlL8+LlN7reEvldtmG62p7Vabi8iR2msN92uzUdRhe7yt4cYGVdPPXMLbdJNl20dDR58/4ofnAjqRQ4vHWN/zaW/8vHnzZOrUqXL99deLiMj8+fPlhRdekEceeUTmzp0bNn7+/Pkh/z9nzhx57rnn5G9/+5sMHz68PXc4HFJQoP/7PxE48wkAAAAAx0B9fX3IzaucMPH5fLJhwwYpKysLycvKymTNmjUR3U8wGJSGhgbJzQ39SMjGxkbp27ev9O7dWy666KKwM6PxxuITAAAAACJ0uO021puISHFxseTk5LTftLOY+/btk0AgIPn5+SF5fn6+VFdXRzTn3/72t9LU1CRXXnllezZo0CBZuHChLF++XBYtWiRpaWkyfvx42bZtWwx758iS9rJbAAAAAEg21r9vsW5DRKSyslKys//zliOPR387lMihS2RDtmFZYZlm0aJFUl5eLs8995zk5f3nLWdjxoyRMWPGtP//+PHjZcSIEfLggw/KAw88EOEjsYfFJwAAAABEKJ6f85mdnR2y+NT06tVLXC5X2FnOmpqasLOhX7ZkyRKZOnWq/OUvf5FzzjnniGOdTqecdtppCT3zyWW3AAAAAJCkUlNTZeTIkbJq1aqQfNWqVTJu3Djj9y1atEiuu+46eeaZZ+TCCy/8yvuxLEs2b94shYWFMc/ZJGnPfDpP6CNOV+hp5+umr1DHTn7lv9R8wPPr1fwPD72n5r/Ye6qauwaeqOY/OOsVNX95+nh9OwNOUPPHD4a/sTh33R517JlpDWp+63t6E3D6SbVq3hjUm0NbTuip5pOy31XzNc4xai79mvR87wF9PqOOU/PMKr0h1Nmju5qn7NWbT00+reul329DjZpv9+nzdNXp+3N/QH9VzFMf3rLZJvpjTdGLd42czXqbq6kdt9nQWJxmeEGvwW9ou3Xo7bL1bfbacSsC+vgebv051RQ0tPU6fWpuasf1uAxtt5ah7dbQXus3VDq6HHqzqraVFEN7bcBwkY/L1F5rGO+MUzuucXyc2mKDpouabG8/1ouj4iyR87H5QryVTM2+0Uj0fJLtuQMgOcTzutsIzZo1S6ZMmSKjRo2SsWPHyoIFC2THjh0yffp0ERGZPXu27Nq1S5588kkRObTw/O53vyv333+/jBkzpv2saXp6uuTkHPr0hl/84hcyZswYKS0tlfr6ennggQdk8+bN8rvf/S7GB2eWtItPAAAAAEg6cbjsVmx+/+TJk2X//v1y1113SVVVlQwdOlRWrFghffv2FRGRqqqqkM/8fOyxx6StrU1uvPFGufHGG9vza6+9VhYuXCgiIrW1tXLDDTdIdXW15OTkyPDhw+X111+X008/PbbHdgQsPgEAAAAgyc2YMUNmzJihfu3wgvKw11577Su3d99998l9990Xh5lFjsUnAAAAAETIsg7dYt1GV8TiEwAAAAAiFM+2266GtlsAAAAAQMIl7ZnPj3+UKc700NbLv3X/TB37/GN6a6Zz6CA1n5C+Wc2nP3WGmqdcrM9xZu6Hav7aBr0xtvp7w9X8yfdGh2UnfrpZHetx6IcsuCFHzS+/crWav+XNVPODA/Qm0KEphpbNnrlqPqJ4p779uno1b+ytvw5y/Ct61Wuwl/5402r0V5Ecbn2/VR/MUvOSlh1qvr1Vb8d1NuhNrHuDGWqeUh/erNpq6W2rKY32rstwNev7wC16262/RT/maQ79mDR4Te2y+vzN7bh6G21zUG/fLXQcVPPWoGH+Tr1919R2m25ox/Vb+n5LNTzegOGVTFM7rtY76zK00Zo6ao3jDdf0mF5rDRruwdSOGzBdM2RoCDVt32GzTdduw2kEn78d0/aTrgEWnVKXO0nS1R4vOhbLEfsPZZf7oT4kaRefAAAAAJBseM9n9LjsFgAAAACQcJz5BAAAAIBIWf++xbqNLojFJwAAAABEiLbb6Nm67Hbu3Lly2mmnSVZWluTl5cmll14qW7duDRljWZaUl5dLUVGRpKeny4QJE2TLli1xnTQAAAAAHDNWjLcuytaZz9WrV8uNN94op512mrS1tckdd9whZWVl8uGHH0pm5qH21HvvvVfmzZsnCxculAEDBsjdd98t5557rmzdulWysvRmUc3zZzwqWVmha+MLt07WB7/9vhpvfSi8RVZEZN6B/mref/FeNQ/+rlnN9wb0ll0roDc6tk5sUPOcl8P3i9OjN4pu9ukNm3kb9WbPy6dtUPMH90xU8/oBeiNnhlNvILWK9PbXr+fqLbtLg3lq3tRbv1/X3jo1bx6Ur+bpe/WfZmeG3jrrP6A3sUpQn09Fk/54rSb9OVLdprfyptSHN6s2GO7T1HYbsPTnmbtJfyXNZWivtVr1NldTs3KTV38upBkaThvb9OdymkN/zta1pat5ZrreRrvbrzc3Zxjaa1sNbboeQ3utz9AS7HHqx8tveE3P1I7rV3ZbqmHbpnZZU9ttwPDXze54u22xTpvttbbLZW2348anTdf0Mxev7cdFgpt6rUQ/1q55IgAAugxbi8+VK1eG/P+f/vQnycvLkw0bNsiZZ54plmXJ/Pnz5Y477pDLLrtMRESeeOIJyc/Pl2eeeUZ+8IMfxG/mAAAAAHCUcdlt9GJqu62rO3RWKjf30Gc9VlRUSHV1tZSVlbWP8Xg8ctZZZ8maNWvUbXi9Xqmvrw+5AQAAAEBSivWS2y586W3Ui0/LsmTWrFnyta99TYYOHSoiItXV1SIikp8feklkfn5++9e+bO7cuZKTk9N+Ky4ujnZKAAAAAIAkFfXi86abbpL33ntPFi1aFPY1x5feIGRZVlh22OzZs6Wurq79VllZGe2UAAAAACDBHHG6dT1RfdTKD3/4Q1m+fLm8/vrr0rt37/a8oKBARA6dAS0sLGzPa2pqws6GHubxeMRjKNcBAAAAgKTC53xGzdbi07Is+eEPfyjLli2T1157TUpKSkK+XlJSIgUFBbJq1SoZPny4iIj4fD5ZvXq13HPPPbYmVhNIleZA6InZ1l8XqWPdE/SF7aPn/1HNb1x6vZr3/2itmj94wj/V/LbKi9VcTu2txv9z8t/V/Im7J4Vl1pAT1LGP79PvMvP93Wo+OCVFzV/9ZICaF56oN/7uCzSpeVOJ3mA8Pv1TNX82Rd83Gb0b1dw6qLfdNhXo28ncozeKOnKy1Tz1gN5karKzobua92jW9/9ufw81dzWGNyXXBfW5pDbpDZttYmjH1Yt3jZzN+gUQKQ59Pt5W/TmVZri6ocGnv7iU4dTbbpsM7biZTr1dutnQXtvDrT9nWy19/qa229agvfF+y9R2a2iwVV75NLfR6lw2G0hdNttinYb5mNhtx3XYnL9p+0HTX3LbjatJ1Ohqdy44skSfaEii49VFu0wAJDlbi88bb7xRnnnmGXnuueckKyur/X2cOTk5kp6eLg6HQ2bOnClz5syR0tJSKS0tlTlz5khGRoZcffXVCXkAAAAAAHDUcOYzarYWn4888oiIiEyYMCEk/9Of/iTXXXediIjcdttt0tLSIjNmzJCDBw/K6NGj5cUXX7T1GZ8AAAAAkJQsR+yXF3TRyxNsX3b7VRwOh5SXl0t5eXm0cwIAAAAAdDJRFQ4BAAAAQFdkWYdusW6jK2LxCQAAAACR4j2fUUvaxef3/jZdnGlpIdkJz69Tx372zKlqfnZ6q5qfsLhev9PhQ/Tx7o1qvvn5wWrunKhv/vJuelXtwo/Cm2Grbxipjv14iz7H0l36HE08H6Sr+aRv642/G73d1byun/4UKnHrTanO7jlqPixfb4vd36g3ljYX6tfJ536gV70Ge+ptt2n79e043PrjOlCXqeY5rXoTa2Vrrr79ppawbH9QPyYpDXrHaXNQb4t1N9n7beZu0feB01AL2ebV902KYXyTT2+jTXPoj6upzTRef7wNgTQ17516QN9+0NC+69KPoV/057LH0Nbrt/TxbkPbrdaO6za0y5o6Z03tuLbHG16GNbXpBg33YGrHDZhe5rW5fbuNovFq00US4ljhMJ4LQIeQtItPAAAAAEg6FA5FjcUnAAAAAETIYcX+sb5J9LHARxWLTwAAAACIFO/5jFr4m40AAAAAAIgzznwCAAAAQKR4z2fUknbxWfpQpbidoa2UTReepo79x/j5an7t9ovU3Nq0Rc0//fVYNV9YX6Tmff9eq+YNv9JbduuCem7528Ky5nF6y2v2Wr1t1eFOUfOP/HojZ68Pwu9TRGRS1ntq/sT+8Wre2E9vo8xw6o2lVn5PNR/f459qvjyoj28p0O/Xtb9BH3/icWruOaBf8+DMyFBzf53elCpBvcm0skWfv9UU3sq7t01v5HU36cewydL3QUqz/pgChvEuQ9uty6FfGGG1GtpfHfqvk1a/nqcZ3uzQ2KbvY1PbbUtAf+6bxh8IdjOM138mWoP6c9nj1Mf7jO24+nMkoFQ0phq27TdcomNqxzW1y5rabk0S3RbrdNrcvr3N22f3jTiGCZl+5mxtP9EP1u72k20+AHAscNlt1LjsFgAAAACQcEl75hMAAAAAkg5nPqPG4hMAAAAAIsXiM2pcdgsAAAAASDjOfAIAAABApGi7jVrSLj6tllaxHKFNgd3+e6c6Ntdw/vbjPw5S87yT9qv5TRc8r+Z3r9FbcwdsfkcfX6o3xs7fP0bNXaUlYdnUoWvUsS8/pLfOOvv3UfNldSPUPPOjvWo+WC8OlVd2DFDzbv3q1LzR0Ozb2jtLzU9L/0zNlzvz1NxVEN4WKyJi1Rnabo/TG4vT9+sNpI4svRE1pVZvMjWpbtIbbDNbasLHtuWoY52NPjVvCOpPfFPbbZvoj9XdosZGzlb9fl2GilOvV39SGZ5q0tJmaq/VG2AbA4Z2XKe+35oN7bU93HrDdKulz8fUduu39OdIiqHt1m+F709je62hCtTlMI3XuQxtqwHDNUAuQxutabzTdpuureG223dNDarBeF3zZHc+iZRMcxERK8nmY1si/23Y0fcN0IU5rNh/hLvqrwAuuwUAAAAAJByLTwAAAACIlBWnm00PP/ywlJSUSFpamowcOVLeeOONI45fvXq1jBw5UtLS0qR///7y6KOPho1ZunSpnHTSSeLxeOSkk06SZcuW2Z+YDSw+AQAAACCJLVmyRGbOnCl33HGHbNq0Sc444wyZNGmS7NixQx1fUVEhF1xwgZxxxhmyadMm+elPfyo333yzLF26tH3M2rVrZfLkyTJlyhR59913ZcqUKXLllVfKW2+9lbDHweITAAAAAJLYvHnzZOrUqXL99dfL4MGDZf78+VJcXCyPPPKIOv7RRx+VPn36yPz582Xw4MFy/fXXy/e//335zW9+0z5m/vz5cu6558rs2bNl0KBBMnv2bJk4caLMnz8/YY+DxScAAAAARMgh/ykdivr2723V19eH3Lxeb9j9+Xw+2bBhg5SVlYXkZWVlsmaNXlK6du3asPHnnXeevPPOO+L3+484xrTNeEjattvtPxgorrS0kOyD0ofUsePfvVbNj1v0rpp/9pNT1Hxmj+1q/uzf9DW6uyBfzSek602P31s1Vs2PGxvejvm97pvVsas/7K3mtecNVPO/bh+m5vmVFWqe4dSbQL0f662t507cpOaf+PWKwIbe+lOu1O1Xc1e23jrbP09vLA426o2lzXn6Mcz6XK96tbIz1Ty11lB96NQbTvc3Zqh5hi/88Vb5uuubbtabgxsMLazuZv3557f07lOX7bZbfR+4Rd8HbT49T3Pox6TZb2iXdejzbwmY2nH151RrUB+famjTNbbXGubjt/TneKqhHVdrsHWb2msNteypLn0ups5Zl6GN1jTe1C4btPTcVBAaNNyD/fZae+PttukmtOHU7va7ZhM/joEu96kPXe3xIr7i+FErxcXFIfGdd94p5eXlIdm+ffskEAhIfn7o2iM/P1+qq6vVzVdXV6vj29raZN++fVJYWGgcY9pmPCTt4hMAAAAAkk6UhUFh2xCRyspKyc7+z0kej0f/CDkREceXXk21LCss+6rxX87tbjNWLD4BAAAA4BjIzs4OWXxqevXqJS6XK+yMZE1NTdiZy8MKCgrU8W63W3r27HnEMaZtxgPv+QQAAACASB3lj1pJTU2VkSNHyqpVq0LyVatWybhx49TvGTt2bNj4F198UUaNGiUpKSlHHGPaZjxw5hMAAAAAInS4NCjWbdgxa9YsmTJliowaNUrGjh0rCxYskB07dsj06dNFRGT27Nmya9cuefLJJ0VEZPr06fLQQw/JrFmzZNq0abJ27Vp5/PHHZdGiRe3bvOWWW+TMM8+Ue+65R77xjW/Ic889Jy+99JK8+eabsT24I2DxCQAAAABJbPLkybJ//3656667pKqqSoYOHSorVqyQvn37iohIVVVVyGd+lpSUyIoVK+RHP/qR/O53v5OioiJ54IEH5PLLL28fM27cOFm8eLH87Gc/k5///OdywgknyJIlS2T06NEJexxJu/i8a/LTkpEV2jI572CpOjZ1Qa6aO7Mb1HzyN15X89da9KuQM1/+SM33XjZUzT/1N6p50cv6m3d3nxvegpnn0ttWAwcP6nMZoW/b8WEPNbe8/1LzfQG9Lbb7x2osEy/7UM3faB6g5k299Xn2cOmtsI4e3dV8RA+9rXeDXz+GrcfpLy+5DzareVsPfT6ptfp2nKl6g2pzfZqaW21K221rjjpWWvS22wMBvQnY3aS3qjYb2m7dLfZeenMb2m5dhvbaYKv+aybFML7FZ2qj1ZtSm9v0hmZT221z0N742oD+XEhzGtp0DS3EHkPbrdamaxobNNQzug3NuwHDoXUaXm4NGNprTe24Jqbtm5jabk3zMfUgmNp07bfjHv023YAVn7knVVNvMuro8weQHOJYOGTHjBkzZMaMGerXFi5cGJadddZZsnHjxiNu84orrpArrrjC/mSilLSLTwAAAABIOsdo8dkZUDgEAAAAAEg4znwCAAAAQISOReFQZ8HiEwAAAAAiZTkO3WLdRhfEZbcAAAAAgIRL2jOfZ6YflOz00LXxnfOvU8fmLX9LzbffrtcEL+/1vJoPeHWqmpe2blHzpkvq1fzuqklq3v2N7Wo+8vYDYdnbXr1J05mVpea9h+9W85aFhWru6q43q77j1ZuDu2/TG1dHePT7/XPVWDVvLfapuUnbcdlqfmrm52q+QUrU3Hec3h7qqNfbfb0n6C3BaYa2W0emoa23wfAjprR47mnVj63Vqu/7/Ya2W1eLqYVVn3uKoe3Wb2jHdXrV2Mjh01/jchpe+/L69X3mMbxAaGq7TTE0wLYE9DZaY3ttm6m9Vh/vU9prjzQfvxX+eN1O01h9n7kNbbSmjlq3oTnYxGVqozW0JTgN8zGPT3Abra3RUXxDV712KhI296WVbO2+AKChcChqSbv4BAAAAIBkw3s+o8dltwAAAACAhOPMJwAAAABEistuo8biEwAAAAAiFYfLbll8AgAAAACOjDOfUUvaxec5G6eIK8MTkh2/4B11rPPEfmp+zbdfVvPNPr35tPgZfXcExg1R8/tOeUrNb3z2ejXvX71WzW867p9hWfnOi9Sx1sDj1fz6PnqD71NbzlPz4Am91fzvB09V89SKGjXv7U5X8w92Fql50fHhzb4iIvsCeutsS6G+/SGp1WruSBmo5pnHNau51dCo329PvbE0c4+hNTdDb7tNqY/8bdX7mjPVvIe3Ts33tuntuM5mvVG4Iag/JnerqSlVz916+a6Rw6vXUaY49Pn4ffrPYapD306LoY02zdh2q7fjpjn09lpvUN9+hlvfz1p7rYi57VZrx00xtNEGDNWepvZa03iXsY1W57T58q7TZgOp7fbaY7T9oOlfColsx7W9bbvju+i/fhIlke27SXasuuhHFAKIUdIuPgEAAAAg6XDmM2osPgEAAAAgQnzUSvT4qBUAAAAAQMKx+AQAAAAAJByX3QIAAABApHjPZ9SSdvFZMN8tbndoy6RzQIk69uPv9VDzFb22qvngf05V874vblLzT341Ss3LMvR2zN4v67m7b7GaD0gJbzl9a73e2tp9hF4vd0FGpZo//dlONT9wqd7g++qOE9W8eM82NTc1lroq0tT8tEHvq/lnbXoDaVOBvv3ehmeus5veGNu/53419zXrLbjeXH0/d/9Ybzi1svX7Ta031AE6wx9XXZPe7Nvdp99njS9bzR0tXjVvCOr72NWiN5+2Wnqzr6vV3m9Ll0/fB05DLWSbXz/mKYbxrW2GdllDW29LQG+vTXHoj7fZsN8KHQf1+RjacdOc+u8Fv4Q/XrdT7531K824RxofMNRRug1tt3pqbse1Pd7Snzum0kxT47KpvTZg2L7pjTWm7Sea3bZeAEmAn1sgLpJ28QkAAAAAyYbCoeix+AQAAAAAO7ro4jFWFA4BAAAAABKOM58AAAAAECkKh6Jm68znI488IsOGDZPs7GzJzs6WsWPHyvPPP9/+dcuypLy8XIqKiiQ9PV0mTJggW7ZsifukAQAAAOBYOPyez1hvXZGtM5+9e/eWX/3qV3LiiYcaUZ944gn5xje+IZs2bZIhQ4bIvffeK/PmzZOFCxfKgAED5O6775Zzzz1Xtm7dKllZWbYm5li/RRyO0NbIfy0cro799dhn1Hz+wX5q3mtRhpo7e+aq+ZUT16j5661qLOlrP1bzfYaG2Qp/Y1hWoN+l7P663mrZw6U/pkB9vZof0Kcijm16g6rl1xtX9wWa1DyrQt/+uCy9NXd9S381by7U6+VynHozrKO7Pv/B2Z+r+Wa94FRae+q/Edx1LWre1l2fT0q9vh1nangjqrdJb1W1Avox3+sz/Ey16m23tUH9OeJu1rfvtfQmUJfheW/i9OrH0OXQX/uyfHqeYhjv9eu/xlIdhhbfgD4+zaG30Zraa1Md+n5rDnrUPMUw3m+Fz8c0NmCoW0wxPNagYbzbtH3DH0KnzXZZUxutiWn7Jna3b7tdNsHzsbdxPQ4Yfj7jtf246egNoR19/gCQZGyd+bz44ovlggsukAEDBsiAAQPkl7/8pXTr1k3WrVsnlmXJ/Pnz5Y477pDLLrtMhg4dKk888YQ0NzfLM8/oi0MAAAAA6FCsON26oKgLhwKBgCxevFiamppk7NixUlFRIdXV1VJWVtY+xuPxyFlnnSVr1hhO44mI1+uV+vr6kBsAAAAAJCMuu42e7cXn+++/L926dROPxyPTp0+XZcuWyUknnSTV1dUiIpKfnx8yPj8/v/1rmrlz50pOTk77rbi42O6UAAAAAODo4Mxn1GwvPgcOHCibN2+WdevWyX/913/JtddeKx9++GH71x1fenONZVlh2RfNnj1b6urq2m+VlZV2pwQAAAAASHK2P2olNTW1vXBo1KhRsn79ern//vvl9ttvFxGR6upqKSwsbB9fU1MTdjb0izwej3g8ekEHAAAAACQVPmolajF/zqdlWeL1eqWkpEQKCgpk1apVMnz4oVZan88nq1evlnvuucf2dusmnyau1LSQ7J9n/1odm2doej350e+oed8VG9W8+nt6m+5Pj1uu5uPf+Z6aFzT8S833TtRbSB/ef0ZY1v3t3erYkpn6e2I/8jWruTMtTc27D9mv5rK0p74dQ1vxh/5MNc/ZrrfjnurRH1f53hFq3lpgqKM1COR2U/Mh6TvVfLP0VnNfrt4G6qjX2319/XLU3GNou3WkK+24jYYfR0Oj6N5W/bFaPn3f1wb0nxNXq76PWw336/aamk8N7bj6097M0HbrNFyo4WvT91uK4YKL1oDeXmtqmPUGDe24TkM7bpu97fssV1jmcerHRGvGFRFxO+2145raZU39qW5Dm66JsR3X8JfW6dS3bxpvt73Wdjuuvc3b/4aEtuN28H/N2NyXlt3HS3stgDiIx3s2O/qv62jZWnz+9Kc/lUmTJklxcbE0NDTI4sWL5bXXXpOVK1eKw+GQmTNnypw5c6S0tFRKS0tlzpw5kpGRIVdffXWi5g8AAAAA6ABsLT737NkjU6ZMkaqqKsnJyZFhw4bJypUr5dxzzxURkdtuu01aWlpkxowZcvDgQRk9erS8+OKLtj/jEwAAAACSEpfdRs3W4vPxxx8/4tcdDoeUl5dLeXl5LHMCAAAAgOTE4jNqUX/OJwAAAAAAkYq5cAgAAAAAugoKh6KXtIvPUf+1SVK7hbZGVgb0j2T53z3j1Lzkj9vV3ErTt1Nw5edq3mrpTZIpf+uu5q7BpWp+86hX1Pz+deeEZQO2v6OOvT7/PTV/una0mjv66m2uV/TdpOYvbRuvb6dI/7icVxtOUnPPjoNq3tulN4G+t6dIzbMLGtS8Mdiq5t7j9EbXQZ4qNRdnXzVOydW3bzXprcLenPDGUhGR1Hq9xdOREd526260dyHCgVb9sWa21qj5/oDejuto0Vtbm4L6fNyt+m/LNtF/Tuy23ToNbbcuQ8Wp36/ve/2ZJuIN6L/2Ug1dry3Gdly9kdYb1Mf3cOtNyVqDrakZ19Re6zL8BfNbhmNobJfVt29qi9VnaZ6PidNmA6mpHdck0e24JsF4XFOVbP86Sbb5dGSJbt7lWAGJw2W3UeOyWwAAAADoJA4ePChTpkyRnJwcycnJkSlTpkhtba1xvN/vl9tvv11OPvlkyczMlKKiIvnud78ru3eHfkTihAkTxOFwhNyuuuoqW3Nj8QkAAAAAETp82W2st0S5+uqrZfPmzbJy5UpZuXKlbN68WaZMmWIc39zcLBs3bpSf//znsnHjRnn22Wfl448/lksuuSRs7LRp06Sqqqr99thjj9maW9JedgsAAAAASSeJL7v96KOPZOXKlbJu3ToZPfrQ2/J+//vfy9ixY2Xr1q0ycODAsO/JycmRVatWhWQPPvignH766bJjxw7p06dPe56RkSEFBQVRz48znwAAAABwDNTX14fcvF6bZRlfsnbtWsnJyWlfeIqIjBkzRnJycmTNmjURb6eurk4cDod07949JH/66aelV69eMmTIELn11luloUHvZzHhzCcAAAAARCqOZz6Li4tD4jvvvFPKy8uj3mx1dbXk5eWF5Xl5eVJdXR3RNlpbW+UnP/mJXH311ZKdnd2eX3PNNVJSUiIFBQXywQcfyOzZs+Xdd98NO2t6JEm7+Ly3cKNkZ4WemD1x0Q/VsVmf6idw8/dtUPMD3x6h5n8/8Tdq/j/VE/Xtr9TbcXd/s5+aX5/zLzV/as2ksMyZmamOHZumvxryg3eGq3neSeGtqiIil2S9q+avV+hNvY3D9dbc12r08WnVe9U8w5mq5s079SbW00duU/PdAb1ns/k4/SldbKhcdaanqXlBD/1VHKulRc293fXawm679SZZKyP8ft0NhupDQ1VnbYs+94w2vYX1YJv+nHJ4fWrerLSwioi4WvWmUb+hFdqpb97I6dMfr9NwoUabqe3WsN9a/Kb2Wv1x+YL6fkgxdL36LUPzsaEdVxtvarvVmnFFRNw223HdhscasAzjDe2yps5Zp+GNLKbxdttlTSWhQcM92G6vtTsfu62liWw5Taa5AF9g+PXSeXW1x9vFOCT2Q3z4+ysrK0MWeB6P/qkc5eXl8otf/OKI21y/fv2hbSt/mCzLUvMv8/v9ctVVV0kwGJSHH3445GvTpk1r/++hQ4dKaWmpjBo1SjZu3CgjRujrqy9L2sUnAAAAACSdOJ75zM7ODll8mtx0001f2Szbr18/ee+992TPnj1hX9u7d6/k5+sfnXiY3++XK6+8UioqKuSVV175ynmNGDFCUlJSZNu2bSw+AQAAAKAz6NWrl/Tq1esrx40dO1bq6urk7bffltNPP11ERN566y2pq6uTcePGGb/v8MJz27Zt8uqrr0rPnj2/8r62bNkifr9fCgsLI34cFA4BAAAAQISS+aNWBg8eLOeff75MmzZN1q1bJ+vWrZNp06bJRRddFNJ0O2jQIFm2bJmIiLS1tckVV1wh77zzjjz99NMSCASkurpaqqurxec79P6pTz/9VO666y555513ZPv27bJixQr51re+JcOHD5fx48dHPD8WnwAAAAAQKStOtwR5+umn5eSTT5aysjIpKyuTYcOGyZ///OeQMVu3bpW6ujoREdm5c6csX75cdu7cKaeeeqoUFha23w435KampsrLL78s5513ngwcOFBuvvlmKSsrk5deeklcLr3vQsNltwAAAADQSeTm5spTTz11xDGW9Z/Vb79+/UL+X1NcXCyrV6+OeW5Ju/i8ZddpktottBl14Lwd6liroVHN90/W3/iaelX4m3BFzKeBX35Bb5Ltt2utmrdN7G7Yvn4PeWv3h29j2ImGbbyu5ikb9bbY/UPUWAak6E2pgT01an5wQD81b/j8OH37DXoTsKkRNXOn/orJyK/r2/mXT7/fluP0Fq9cl94c5szS91u/7PBjIiKyx6e315rabl31etVrsFt4C3FKkzpUHG69nbWlWX9Mll9vVd3n1x+ro1WfY0NQf46Y22713OW199KeU9/F4jT0ygX9+s9VikPPfQFTO64+/5aAvv9TDQ2zrUFTm64+vikYfhxTnPoxNLXXmrYdtPR94DQ81qDNdlwTu+2yxnZcwx9C03i78wkYtm8qAzS16dpvx418vO0mXRxbHC+g60jgmcvOLGkXnwAAAACQbOLxns1Evecz2fGeTwAAAABAwnHmEwAAAAAiFcfP+exqWHwCAAAAQIS47DZ6XHYLAAAAAEi4pD3zueWRk8X1pUbWHs1b9cEp+sM4/vpP1Py3fZep+fWfXa7m/ZbrbbrOoYPU/OdD/qHmf6o/Qc2DH38WllXfcro69m2vXqWXt8Gr5juv16tDg4Zz/Vab3rLZMEDP0ytS1dxU0VgVaFHzbpV6i+QpaXrD8dvN+r5szdMfl8ehN5Ba2XoD7KDM7Wq+JxjeUisi4utuaOVs1B+vvygnLEtpNDRvpupzb2s2/PgG9ebTA75sNbd8etttvant1qtv3294Trn0p6YEDO24TsNz3GVorxVD263LUDvp9ev7LcUwf3M7rv4z4Q0atm8Y77cylLGGfWzp23Y79X3pE3vNvqY2XVM7bsDwqq1pPuZ22cS1xR4ab2u4bUlVcGqYjOnnzfZL74l+sEm1MwHAgMtuo5a0i08AAAAASDZcdhs9Fp8AAAAAECnOfEaN93wCAAAAABKOM58AAAAAECnOfEaNxScAAAAARIj3fEYvaRefWX95R9xfaiit+J+x6tiUBn0b60+4X81NjZGf/0VvUM175y0133H7aDW/PPOgmg949TI1L3VsCcvaxtarY/9Qc5aap3+wU83PP/GAmr/n09s0nVlZat7vhD1q3vJGoZq7cvRm1Q99PdW82069ErU0Rd+XDx/sq+b+4/R2X5NA9/CmURGRE9Oq1Xy1lKh5W3e9ydTR3KrmvqxeYVmqqe02zWPYtt5kanLAm6l/wafPscHQ7Ov06o+11dBk6vbqualx2WnvEIrD0HbrNLyroC2g5ymGls3WgN42bGqkNbXdphobbMOPo2nbPmXskcYHLf2xup2m8fpOMLXXGvpTxWnz5VyX4S9wwLAdU9ut3fEmdsfbZqfRtav+6yRSNttxrWRr9wWALiZpF58AAAAAkHS47DZqLD4BAAAAIEIOyxKH4YovO9voimi7BQAAAAAkHGc+AQAAACBSXHYbNRafAAAAABAh2m6jl7SLT2v0ULHcaSHZPd9ZqI79Z+MANX/Lq7dU/qnm62pe9P8+1efSI0fNB07apub7gy1q3vOlNDV3DOwflt140mp17G/+eb6aD6her+ZX5G5W8/+rPV2fS+8CNb+gcKOar6roruZSmKfGa5tOVPOUqlo1z3elqvm2feFtsSIi3Xs1qnljUG909fbUj0m/lH1q7nCXqnlqd72t12rWnwu+7PAr3lPr9e5QR5o+R3ezvavma1v19tosn96sXBvQm4AdrXrbbVNQn4/LZ2q71R+vy6fGRg6/Xkfpcui53683xpq6g70BQ3utYf6mttsUh77fvMHw31MZbn0nmJq6TW23AUNVZ4pDn7tpvNPYLmvY94Z2XH2W5u2bOG02kNpvu03s9jWm9mfbku1fM8k2n44s0c27HCsAR0HSLj4BAAAAIOlw2W3UWHwCAAAAQIS47DZ6tN0CAAAAABKOM58AAAAAECkuu41a0i4+9830iutLfSdfTz+gjr0kc7Oalyy/Qc27faI/7OP3va3m9VeMUvOn+/5GzX+772tqftxru9R8z7nHh2VXZ21Vxz6+8SI1dxpKaUal6sUl//XZEDXvdWKmmp/XbYuav1Y5WM2bhxSq+bp9JWruqtmv5hlOvXCopbqbmg8eppdG7QnoZS+tuXrNTJFLLxBypOrz6ZWjFx1ZLXrhkDc7vDkic7c+Ryvdo+buRkP7hKEtpaFV3063Nv1+D7bpzwWHV39OeS1DkY/XUD5j+HBlp83CIafPUJJjuLAjGNDnmWIqKDKO1x+Xz1Q4ZKjb8Sv7LdVQTqSNPTQX07b1uThtFg65TeMtewVF+lbsj49Hwc+h7RtKvuxuP5GFRgkvmUnweCBKhl8vQFLgstvoJe3iEwAAAACSDmc+o8Z7PgEAAAAACceZTwAAAACwoateNhsrFp8AAAAAECnLOnSLdRtdEJfdAgAAAAASLmnPfL546iLJzgpdG49e/3117I8Hv6TmAx9rUnPXgXo1b554qprXXNKq5r3d6Wr+f2tPV/MB2/U23QPjjwvLsp16e+1xG/RWVTmxnxpnONepeWBLtpofHKDXyw1M0Vs2g3v3qXldSbGeV4U/VhGRExv0JuCApbdRplXp8xnytSo1396Wo+atufrrL7mGll1Hhn7Mi7Nq1fygz6/mfqXt1t2o17xamfpzwa0X6YrDpe+bltYU/RsCelNqXZv+WB1+vYm1ydK37zS03foN7a8un83mUH064jTUcgb9+jF3Gcb7bLbdegP6r9VUYyNt+PadhhZWU3uty2Z7rakdN2jp+8bUjmtiaq81sdsua2zHNbyKbLvQ1bB9U0OzialNN5HXatlq0gWQHPi57XBou41e0i4+AQAAACDp0HYbNS67BQAAAAAkHGc+AQAAACBCjuChW6zb6IpYfAIAAABApLjsNmoxXXY7d+5ccTgcMnPmzPbMsiwpLy+XoqIiSU9PlwkTJsiWLVtinScAAAAAoAOL+szn+vXrZcGCBTJs2LCQ/N5775V58+bJwoULZcCAAXL33XfLueeeK1u3bpWsrKyIt7+quadkfKm1s+gefbpzL75czfttWqvmbYY6wM/v0ptYfz3qWTV/tLa/mh//sr5913H69q86ZX1Y9k+voZHzX5+r+f5vDFHzCr/ejpu7RX+5pepsvQXT49CbTIOtehNwQz81FtdOvbnV9FlHB4N6pWtGlT5+SPpONf+Xt0jNvT316WSY2m67Zap534wdan7QcEmFLzt8/s5mve22rbveOutuMjR7uvWfkzZD261laLut9WeouXj1eTYE9Xm6TG23hiZjp755I6ff8PPm0H+GrDZTA6w+3temt926DC9Z+oKmdly9lrc1GH5cTM24zUGPYdumJl39uWBqizW34+rHKmgY7zbMJ2B4ldc4H1N7bZzaceMlkQ2zdh+r/TvQY1PTeLy2HzcduSW0I88d6OKSve324MGDcvPNN8vy5ctFROSSSy6RBx98ULp37278nuuuu06eeOKJkGz06NGybt1/PjnD6/XKrbfeKosWLZKWlhaZOHGiPPzww9K7d++I5xbVmc/Gxka55ppr5Pe//7306NGjPbcsS+bPny933HGHXHbZZTJ06FB54oknpLm5WZ555plo7goAAAAAkodlxeeWIFdffbVs3rxZVq5cKStXrpTNmzfLlClTvvL7zj//fKmqqmq/rVixIuTrM2fOlGXLlsnixYvlzTfflMbGRrnoooskYDiRoYlq8XnjjTfKhRdeKOecc05IXlFRIdXV1VJWVtaeeTweOeuss2TNmjXqtrxer9TX14fcAAAAAAD2fPTRR7Jy5Ur5wx/+IGPHjpWxY8fK73//e/n73/8uW7duPeL3ejweKSgoaL/l5ua2f62urk4ef/xx+e1vfyvnnHOODB8+XJ566il5//335aWXXop4frYXn4sXL5aNGzfK3Llzw75WXV0tIiL5+fkheX5+fvvXvmzu3LmSk5PTfisuLrY7JQAAAAA4Kg5fdhvrTUTCTsJ5vd6Y5rZ27VrJycmR0aNHt2djxoyRnJwc48nAw1577TXJy8uTAQMGyLRp06Smpqb9axs2bBC/3x9ykrGoqEiGDh36ldv9IluLz8rKSrnlllvkqaeekrQ0w3v3RMTxpTfAWJYVlh02e/Zsqaura79VVlbamRIAAAAAHD1WnG4iUlxcHHIiTjvBZ0d1dbXk5eWF5Xl5ecaTgSIikyZNkqefflpeeeUV+e1vfyvr16+Xr3/96+2L4erqaklNTQ15y6XIkU8yamwVDm3YsEFqampk5MiR7VkgEJDXX39dHnroofZTudXV1VJYWNg+pqamJuxs6GEej0c8Hr1EAwAAAACSSTwLhyorKyU7O7s9N62LysvL5Re/+MURt7l+/aESU+2k35FOBoqITJ48uf2/hw4dKqNGjZK+ffvKP/7xD7nsssuM3/dV2/0yW4vPiRMnyvvvvx+Sfe9735NBgwbJ7bffLv3795eCggJZtWqVDB8+XEREfD6frF69Wu655x47dyV3L5osLk/o2dXit/T22hP39FHzlnNGqrnLr7f4PTr+z2peluFX85LlV6n54Dcr1LzhDL0dd1rusrDsR9v1Bt9A/R413zdS/wn4e6PegpuzpVbN06c1q3lVm96aa2pWdZXo49NX643HDsMP2s42ffvdqvXm0AEpNWr+Wt1gNffmRv4GaRGRYLbeAFvi2avmm0VvOPZnhR8vR7PeHNzWO1vNU/RDJY5UvanXatFbWE1veK/z61c3WH7956E+qI93+vV93Gq4X5ff1Hwan3ZcadMv+HAaLgTxB0zttfrmfUHDz4ShHbctGH6/dttrTeMDlv6YjOMN+8BuO65pvKk/1W3zk7bttteaGmMDhmNit73WbiOtrc3bbURNeDtuB/9gOtp3ASSZ7OzskMWnyU033SRXXaWvPQ7r16+fvPfee7JnT/h6Ye/evcaTgZrCwkLp27evbNu2TURECgoKxOfzycGDB0POftbU1Mi4ceMi3q6txWdWVpYMHTo0JMvMzJSePXu25zNnzpQ5c+ZIaWmplJaWypw5cyQjI0OuvvpqO3cFAAAAAMknHm21Nr+/V69e0qtXr68cN3bsWKmrq5O3335bTj/9dBEReeutt6Surs7WInH//v1SWVnZfjXryJEjJSUlRVatWiVXXnmliIhUVVXJBx98IPfee2/E2436cz5NbrvtNmlpaZEZM2bIwYMHZfTo0fLiiy/a+oxPAAAAAEhGyfw5n4MHD5bzzz9fpk2bJo899piIiNxwww1y0UUXycCBA9vHDRo0SObOnSvf/OY3pbGxUcrLy+Xyyy+XwsJC2b59u/z0pz+VXr16yTe/+U0REcnJyZGpU6fKj3/8Y+nZs6fk5ubKrbfeKieffHLYJ6AcScyLz9deey3k/x0Oh5SXl0t5eXmsmwYAAAAA2PD000/LzTff3N5Me8kll8hDDz0UMmbr1q1SV1cnIiIul0vef/99efLJJ6W2tlYKCwvl7LPPliVLloScQLzvvvvE7XbLlVdeKS0tLTJx4kRZuHChuFyGt3Yp4n7mEwAAAAA6rS+01ca0jQTJzc2Vp5566sh3/4XLftPT0+WFF174yu2mpaXJgw8+KA8++GDUc2PxCQAAAAARSubLbpNd0i4++z72obgdoa2d+78zRh3bY9F6Na+6V2+Oat2fruamVtsNXr1Os+9yQ4PiHr1xdec5/dS8JKVbWLblLb0Zt7S73og68JQdav6XnSPUPLNC/zzVCwuq1Pwtb4GaO3vmqvmo3vr2d+wcoOauHt3V/D3v8WqeVq1XvRa59Rbcj2r1di9nT/3Y+i29DbQtR3/u9Evdp+biCP+cJRERKzv8uWa16MfW383QttpsaAg1NAe7mm19rK/UefXH6vLVq3lDQB/vbNWPidfwS9dlaK8NGl4idOqbN3L49TpKl6HitK1N3/+mvekztuPqzylvMEUZqz8ov6Vv2+PUf3eZxqc49bkETe24xvH22m5NzG20hvGG3NxeG5/5JBPTz0OHl2T73kqy+diS8GbfDrxvABwzSbv4BAAAAICkE7QO3WLdRhfE4hMAAAAAIpXk7/lMZvauwwMAAAAAIAqc+QQAAACACDkkDoVDcZlJx8PiEwAAAAAiZVmHbrFuowtK2sWnIztLHM7Q1s6xM/VW27d9o9R8xem/UfP1hgbV39UWq/kzO05T8+zXtqi5s18fNb9y7Ntq/pEvvLm1YK3eZNo2uJ+az+i9VM1//PfvqPmJTRVqfk63D9X8/upz1Nwq7KXmZ/d4Rc3/slPf98GCnmq+samfmrtq6tS8hzNNzXft667mvXo0qHlj0Kvmvu7hzaQiIkVufT4Otz4+I1tptjW03foy9avj0/fpjaiONEPbbau919jqffp2urfp99sQ1Pe9w693lrYamlidfv2XcVD0nwlD0auRo83Q0Gp4F0KgTc9TDe24/qCp7Vafv19pmHUZx+rb7ubQnzsBw2NyGt5o4jMdE8P4gOF1W1Pbrd3xJvbH2xoeRTtu4rZvd9u2X0q3+9J9oucDRMlQvg0gySTt4hMAAAAAkg2f8xk9Fp8AAAAAECnabqPG4hMAAAAAIuSwLHHE+J7NWL+/o+KjVgAAAAAACceZTwAAAACIVPDft1i30QUl7eJz6w+PF2d6aHvmPwr/ro4994ZCNU8zNJ9d2U1vJj1x0bfVPOsz/QRxVmCXmu+doDe6/rjXIjUvr54YlmW/XamO3X1pPzU/J32fmvf4wNAumZmp5gNT9LbLNz/vr+Z5JelqPi79MzV/tnqEmjedqu+zdw/oeVrtATVPcejzb9urN7H266Pv571B/VKI1u769o9zGppnU/W2257dwhuOLZ9PHevvpsaSvUO/TystVc1dzYYfCEOdZpNX305OQG+vbQzo+1h8eh2tqe3W5TU1ver3a7ft1um3V4kYCJgaY3XeNv3XqsvUMBsMH58i+mM1td2mOvTngml8ikPfftDUjmto3zW117pN4w11lKb2WtPfZbvjbbfXGnJT47Ld7XfZlgkA8UGz7zHHZbfR47JbAAAAAEDCJe2ZTwAAAABIOrTdRo3FJwAAAABEyrIO3WLdRhfEZbcAAAAAgITjzCcAAAAARMhhxd4d11W755J28fnUBY9It6zQE7M/2DlBHbts4F/U/MyNU9X8xwNfUvMTFzWquWt/g5q3nDFUzfdPbFXznk69Gfb5DcPCsgG73lbH1o4sUvN0h95MmvtheKuqiIijX2819zj0dlbZprfj1pXolWv9U/TtBPfrLbX1xX31vCZXzU9o2qnmAUtvo0zbq7d+Duy2R813t2Wpube7/niznPqPkiNdb4AtyKwPy+oMrbD+LP0+Xc2Gttt0/bng1p+W4nDp+6a11fBcMLTd1rXpz2+HX59nq6XvM6fP1Kyq/5Z2+Ww2merTEaehPtBq0y8QcRnGtwX18SmGBlhvIHw/pBraaE3ttU5DC6vfsI9dNttrje24lr12XBNTe62J3XZZYztunC55Ms0nYGP7pibdRP/rxFB2DQA4Ei67jRqX3QIAAAAAEi5pz3wCAAAAQLJxBA/dYt1GV8TiEwAAAAAixWW3UeOyWwAAAABAwnHmEwAAAAAiZf37Fus2uqCkXXz2cPklyxV6Yva9+05Rx34y5zU1z34sW81/duHlaj7gHb1h1lCOKdt/VKDmPxqxUs2XNvVQ87w14Q2Wrmx97ucM+UjNt/h9au7eprfC1p19oppXtemNv90/VmPZO0pvwTS15gZb9crVJr18V6TKo+dB/X4bLa+ap9foP+EnePS22099eWruy9GnY2obdmRmqHlx+q6wrC5oaCzVi4bF2awf80CWvs9cLfp2HG7910CbV88tQ9ttfZve7CuGttumoGGeflNzq5479ZJgI6fhB9rl0C8Esdr0OlCnoSbUHzC14+rPwTalMdbUFusP6m23pnbcZsM+NrXXBozttYY2V0M7rsvULmsY7zTsm4DhD7NxPoZLmOLVjhsviWyYtftY7d+BHpuaxuO1/bjpyO2+HXnuQCfjsCxxxHjZbKzf31El7eITAAAAAJIO7/mMGu/5BAAAAAAkHGc+AQAAACBSlojE+m6Drnnik8UnAAAAAESK93xGj8tuAQAAAAAJl7RnPi94dYY400PbMwcsWqeO/WbZDDUfsOIdNT+xTm/NdQ3UG2BNbwi+fsJraj41Z5uaj15/nZoXr6sJy/zDT9C3fdwf1PyJ/ePUPLBvv5rvHzpQzde0Fql5zrZmNfdfobfXNgb13FTz6C/WW2oz39cbVE0NrXsC+jUQ6fv0/ITU8H0vIvLX2pFq7uuub8fYlJqZrubHe2rDsg9Ebzhu66bfp6NF32dt+d3U3N1ieIUtRW8mtlr1ZlXTz0ODXz9Wlk+vo2229CZWh8/Udqvfr6nt1tS+6fTbrIxsM7XXmtpu9f1meqXPFwh/LqeK3kbrDerPe2M7rhWf8cZ2XMOjchu2b2rHNc3HxG4brd3xdhtj7bbX2tl+hy84TXT7bqIl8gB0+IMLdGGWxKFwKC4z6XCSdvEJAAAAAEmHttuocdktAAAAACDhOPMJAAAAAJEKSuyXzsfalttBsfgEAAAAgAjRdhs9LrsFAAAAgE7i4MGDMmXKFMnJyZGcnByZMmWK1NbWHvF7HA6Hevv1r3/dPmbChAlhX7/qqqtszS1pz3wOnF8nbldoY6r/a6eqY0sXtKm566QB+sb/+a4aV/x8rJqnNuib+XHPxWruN7Rsul/uruaBT94Ky3ZfPlodO1IvCJVrPtIbfEvd76u5c0i9mv/9gL6dlIpqNR9XWKvmW/366xrObnoTa98ivZW35cVCw3Yy1fwzf66ap9f41LzIpbf4ftrYS80DPfTnmkmgm37Aeqdqj1dvuw1005tGxas/prYMvW01xdB260jV224dXnuvTdUb2m6lTW8+bgrq+8bp1x+vodRWXH79cQUNNXIOe4dQHAFDQ6vhtbtAQM9TDJfn+ILhx8vcRqsf21RjG61+p+b2WlMbrWEfW4afc8P8g5a+fbfTMF5NRZw2KwJNV0YFTM8Rw+O1O/6YsHsZWDLNvROw2J9A15DkhUNXX3217Ny5U1auXCkiIjfccINMmTJF/va3vxm/p6qqKuT/n3/+eZk6dapcfvnlIfm0adPkrrvuav//9HT9kx1MknbxCQAAAABJJ4kXnx999JGsXLlS1q1bJ6NHHzqZ9fvf/17Gjh0rW7dulYED9Y9bLCgoCPn/5557Ts4++2zp379/SJ6RkRE21g4uuwUAAACAY6C+vj7k5vXqn+MeqbVr10pOTk77wlNEZMyYMZKTkyNr1qyJaBt79uyRf/zjHzJ16tSwrz399NPSq1cvGTJkiNx6663S0GC4RNSAM58AAAAAEKk4nvksLi4Oie+8804pLy+PerPV1dWSl5cXlufl5Ul1tf42ui974oknJCsrSy677LKQ/JprrpGSkhIpKCiQDz74QGbPni3vvvuurFq1KuL5sfgEAAAAgEjF8aNWKisrJTv7P70fHo/eiVFeXi6/+MUvjrjJ9evXi8ih8qAvsyxLzTV//OMf5ZprrpG0tNA+j2nTprX/99ChQ6W0tFRGjRolGzdulBEjRkS0bRafAAAAABCheH7USnZ2dsji0+Smm276ymbZfv36yXvvvSd79uwJ+9revXslPz//K+/njTfekK1bt8qSJUu+cuyIESMkJSVFtm3b1vEXn8EduyToCG3h3P/rfurY4y7Zqub/un+Mmg++r1jNL/mmfh30xgP6+AMB/ZrsJQ1D1bzwlX1qHkwJPwypYw7oYw39j5kb9aYpV+8iNb+w/xY1f+7jk9W8pEYff053fd+/0aw3DTt76W20p/f6XM3fqdJbZx25PdT8w9bj1Txlf5Oa57r09tDttYbW3By9ubU5qDfP+rNT1bzAXRceOvW5uLrp9axWqz4Xf6b+Vm53i/7ccaTobbeuFntvCW/y6481ra1RzRsCejuuw6c/3lZDU6qzzdR2qz9ep822W2nT79dlePXQ1HarH10Rv9J2m2qYe5uh7dZpGG9qx01z6N3BpvZal6G91tSO67LZ+Glq0zVt39xGGx/OWF/N/goRvvD877HxKaQwtT/blmxtrsk2n44swc97jhUQm169ekmvXvq/i79o7NixUldXJ2+//bacfvrpIiLy1ltvSV1dnYwbN+4rv//xxx+XkSNHyimn6J+A8UVbtmwRv98vhYX6p1NoKBwCAAAAgEgdfs9nrLcEGDx4sJx//vkybdo0Wbdunaxbt06mTZsmF110UUjT7aBBg2TZsmUh31tfXy9/+ctf5Prrrw/b7qeffip33XWXvPPOO7J9+3ZZsWKFfOtb35Lhw4fL+PHjI54fi08AAAAAiFTQis8tQZ5++mk5+eSTpaysTMrKymTYsGHy5z//OWTM1q1bpa4u9Eq8xYsXi2VZ8u1vfztsm6mpqfLyyy/LeeedJwMHDpSbb75ZysrK5KWXXhKX4UpCTdJedgsAAAAAsCc3N1eeeuqpI46xlDOvN9xwg9xwww3q+OLiYlm9enXMc2PxCQAAAACRiuNHrXQ1Sbf4PLwKb7PCCzECzXrBjzZWRCTYoheytAX17Xgb9e20NenjGxr0Eo7WRr3RpM1QUBS08VjrDfcZ8MbnsQabDdsx7OPmBr3io9Vv2Ad2932bvcdld98b96dh/wd8+uM1bcc0/yZlvPF5bDomhpKjNr8+XvyGORq2EzQUGpnmafo5abP07bfYPFaNpn1seLx2f1ZM402/R8zPHX286feFtt9Mj9XXqO/LZpfh59BQ3uQ2tC6lGrbjbdWPeZOljzfNs9FjKEZqMoy3sc9E7O3jI423/fvX9vjw54jx+Wd4PsVtvM3nN+PjNz6Z5tIlxxv+xh2L8ck0l0SPr288lGln3TqWeLxns6Pvg+g4rCQ7+jt37gz7sFUAAAAAnUNlZaX07t37WE/Dtvr6esnJyZFz+t8sbqf+eZyRagt65aXPHpC6urqIPmqls0i6M59FRUVSWVkpWVlZ0tDQIMXFxWEfvoqOr76+nmPbiXF8Oy+ObefFse28OLadV0c7tpZlSUNDgxQV6R8F2GFw2W3Ukm7x6XQ6218Jcfz7w9Ai/fBVdDwc286N49t5cWw7L45t58Wx7bw60rHNyck51lOIXdCSmC+bTWDbbTLjo1YAAAAAAAmXdGc+AQAAACBpWcFDt1i30QUl9eLT4/HInXfeKR5PbG/oRfLh2HZuHN/Oi2PbeXFsOy+ObefFsT1GeM9n1JKu7RYAAAAAkk172+3x0+PTdrvr0S7Xdst7PgEAAAAACZfUl90CAAAAQFLhstuosfgEAAAAgEhZEofFZ1xm0uFw2S0AAAAAIOE48wkAAAAAkeKy26ix+AQAAACASAWDIhLj53QGu+bnfHLZLQAAAAAg4TjzCQAAAACR4rLbqLH4BAAAAIBIsfiMGpfdAgAAAAASjjOfAAAAABCpoCUxf1BnsGue+WTxCQAAAAARsqygWFZsbbWxfn9HxWW3AAAAAICE48wnAAAAAETKsmK/bLaLFg6x+AQAAACASFlxeM8ni08AAAAAwBEFgyKOGN+zyXs+AQAAAABIDM58AgAAAECkuOw2aiw+AQAAACBCVjAoVoyX3fJRKwAAAAAAJAhnPgEAAAAgUlx2GzUWnwAAAAAQqaAl4mDxGQ0uuwUAAAAAJBxnPgEAAAAgUpYlIrF+zmfXPPPJ4hMAAAAAImQFLbFivOzW6qKLTy67BQAAAAAkHGc+AQAAACBSVlBiv+y2a37OJ4tPAAAAAIgQl91Gj8tuAQAAAAAJx5lPAAAAAIhQm+WN+bLZNvHHaTYdC4tPAAAAAPgKqampUlBQIG9Wr4jL9goKCiQ1NTUu2+ooHFZXveAYAAAAAGxobW0Vn88Xl22lpqZKWlpaXLbVUbD4BAAAAAAkHIVDAAAAAICEY/EJAAAAAEg4Fp8AAAAAgIRj8QkAAAAASDgWnwAAAACAhGPxCQAAAABIOBafAAAAAICE+//HJf0qcksSngAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1228.8x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def getPositionEncoding(seq_len,dim,n=10000):\n",
    "  PE = np.zeros(shape=(seq_len,dim))\n",
    "  for pos in range(seq_len):\n",
    "    for i in range(int(dim/2)):\n",
    "      denominator = np.power(n, 2*i/dim)\n",
    "      PE[pos,2*i] = np.sin(pos/denominator)\n",
    "      PE[pos,2*i+1] = np.cos(pos/denominator)\n",
    "\n",
    "  return PE\n",
    "\n",
    "PE = getPositionEncoding(seq_len=50, dim=128, n=10000)\n",
    "print(PE)\n",
    "\n",
    "caxes = plt.matshow(PE,interpolation ='nearest')\n",
    "plot.colorbar(caxes) \n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## transformer 中的位置编码的使用\n",
    "\n",
    "https://blog.csdn.net/qq_52785473/article/details/124537101\n",
    "\n",
    "* register_buffer(self, name, tensor)是一个PyTorch中的方法，它的作用是向模块（module）中添加一个持久的缓冲区（buffer）缓冲区是一种不被视为模型参数（model parameter）的张量（tensor），它不会在训练过程中更新梯度（gradient），但是会作为模块的状态（state）被保存和迁移"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### classwork2\n",
    "\n",
    "* 已知如下类，通过给定的x运行下面的词嵌入类和位置编码类，并打印结果张量及其维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "#词嵌入\n",
    "class Embeddings(nn.Module):\n",
    "    def __init__(self, d_model, vocab):\n",
    "    # d_model:词嵌入维度\n",
    "    # vocab:字典大小\n",
    "        super(Embeddings, self).__init__()\n",
    "        self.lut = nn.Embedding(vocab, d_model)\n",
    "        self.d_model = d_model\n",
    "    def forward(self, x):\n",
    "        return self.lut(x) * math.sqrt(self.d_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 4, 512])\n"
     ]
    }
   ],
   "source": [
    "d_model = 512  # embedding_size\n",
    "vocab = 1000  # 词典大小\n",
    "x=torch.tensor([[100, 2, 421, 508], [491, 998, 1, 221]], dtype=torch.long)\n",
    "emb = Embeddings(d_model, vocab)\n",
    "embr = emb(x)\n",
    "print(embr.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "#位置编码\n",
    "class PositionalEncoding(nn.Module):\n",
    "    def __init__(self, d_model, dropout, max_len=5000):\n",
    "    # d_model:词嵌入维度\n",
    "    # dropout:置零比率\n",
    "    # max_len:每个句子最大的长度\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "        pe = torch.zeros(max_len, d_model)\n",
    "        position = torch.arange(0,  max_len).unsqueeze(1)\n",
    "        div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(1000.0) / d_model))\n",
    "        pe[:, 0::2] = torch.sin(position * div_term)\n",
    "        pe[:, 1::2] = torch.cos(position * div_term)\n",
    "        pe = pe.unsqueeze(0)\n",
    "        self.register_buffer(\"pe\", pe)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.pe[:, :x.size(1)]\n",
    "        return self.dropout(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 4, 512])\n"
     ]
    }
   ],
   "source": [
    "dropout = 0.1\n",
    "max_len = 60\n",
    "pe = PositionalEncoding(d_model, dropout, max_len)\n",
    "pe_result = pe(embr)\n",
    "print(pe_result.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('pe',\n",
       "              tensor([[[ 0.0000,  1.0000,  0.0000,  ...,  1.0000,  0.0000,  1.0000],\n",
       "                       [ 0.8415,  0.5403,  0.8268,  ...,  1.0000,  0.0010,  1.0000],\n",
       "                       [ 0.9093, -0.4161,  0.9302,  ...,  1.0000,  0.0021,  1.0000],\n",
       "                       ...,\n",
       "                       [ 0.4362,  0.8999, -0.8753,  ...,  0.9982,  0.0585,  0.9983],\n",
       "                       [ 0.9929,  0.1192, -0.0926,  ...,  0.9981,  0.0596,  0.9982],\n",
       "                       [ 0.6367, -0.7711,  0.7711,  ...,  0.9981,  0.0606,  0.9982]]]))])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pe.state_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自注意力机制代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class SelfAttention(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size):\n",
    "        super(SelfAttention, self).__init__()\n",
    "        self.query = nn.Linear(input_size, hidden_size)\n",
    "        self.key = nn.Linear(input_size, hidden_size)\n",
    "        self.value = nn.Linear(input_size, hidden_size)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 计算Q、K、V\n",
    "        q = self.query(x)\n",
    "        k = self.key(x)\n",
    "        v = self.value(x)\n",
    "        # 计算自注意力矩阵\n",
    "        attn_weights = torch.bmm(q, k.transpose(1, 2))\n",
    "        attn_weights = F.softmax(attn_weights, dim=-1)\n",
    "        # 使用自注意力矩阵对V进行加权平均\n",
    "        attn_output = torch.bmm(attn_weights, v)\n",
    "        return attn_output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 8.3025e-01,  7.7278e-01,  7.3960e-01,  ..., -3.1352e-01,\n",
       "           1.7342e-01, -7.2156e-01],\n",
       "         [-5.2339e-02,  2.8183e-01,  2.1247e-01,  ..., -3.9275e-01,\n",
       "           1.1995e-01,  5.8236e-01],\n",
       "         [ 2.7860e-01,  4.8669e-01,  1.6767e-01,  ...,  3.8547e-01,\n",
       "          -6.0063e-01, -2.1536e-01],\n",
       "         ...,\n",
       "         [ 1.3347e-01,  2.2957e-01, -4.2782e-02,  ...,  2.3184e-01,\n",
       "          -1.3142e-01, -5.2254e-02],\n",
       "         [-1.3501e-02,  1.0049e-01,  1.2684e-01,  ...,  2.7119e-01,\n",
       "          -1.9974e-01,  1.4608e-01],\n",
       "         [-2.8918e-02, -3.0294e-01, -1.6875e-01,  ...,  7.6529e-02,\n",
       "           5.7324e-02, -6.7287e-01]],\n",
       "\n",
       "        [[-9.5269e-02, -1.7231e-02, -7.3497e-01,  ..., -6.3840e-01,\n",
       "          -2.0553e-01, -3.1382e-02],\n",
       "         [ 3.8260e-01,  7.8272e-01,  2.7927e-01,  ...,  1.1539e-01,\n",
       "          -5.8368e-01,  6.6879e-01],\n",
       "         [ 3.1963e-01, -6.7354e-01, -4.7888e-02,  ...,  1.3886e-01,\n",
       "          -5.6695e-01,  4.7908e-01],\n",
       "         ...,\n",
       "         [ 4.8869e-01, -4.3439e-01,  5.0466e-02,  ...,  5.1602e-02,\n",
       "          -4.6732e-01,  3.4571e-01],\n",
       "         [ 2.5192e-01,  2.6341e-01, -9.5167e-01,  ...,  8.8073e-02,\n",
       "           8.4320e-02,  3.1297e-01],\n",
       "         [ 4.7435e-01, -5.4737e-01, -8.6699e-01,  ...,  3.0750e-01,\n",
       "          -2.6796e-01,  3.0907e-01]],\n",
       "\n",
       "        [[ 8.9739e-02,  2.6811e-01, -3.0982e-01,  ..., -1.8338e-01,\n",
       "          -3.8594e-02,  1.5753e-01],\n",
       "         [-1.5329e-01, -1.7685e-01, -9.8341e-02,  ..., -4.4869e-01,\n",
       "          -2.0917e-01, -5.7830e-01],\n",
       "         [ 1.2280e-01,  1.5603e-01, -1.5913e-01,  ..., -2.9564e-01,\n",
       "          -3.4006e-01, -1.9447e-01],\n",
       "         ...,\n",
       "         [-6.0711e-01, -8.7723e-01, -8.6103e-01,  ..., -4.2362e-01,\n",
       "           1.0832e-02, -5.0467e-01],\n",
       "         [-5.0899e-01,  2.0693e-01, -3.9636e-01,  ..., -4.5236e-01,\n",
       "          -5.1858e-02,  1.2173e-01],\n",
       "         [-1.8101e-01,  5.9718e-01,  1.3920e-01,  ..., -4.5508e-01,\n",
       "          -1.9719e-01, -1.4848e-01]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[ 1.3279e-01, -2.3987e-01, -2.0989e-01,  ...,  4.4495e-01,\n",
       "          -2.9491e-01,  1.6682e-01],\n",
       "         [-1.2641e-01,  1.8780e-01,  1.0200e-01,  ..., -7.3079e-01,\n",
       "          -5.7564e-02,  1.5786e-01],\n",
       "         [ 6.7600e-01,  8.0135e-01, -5.8610e-01,  ..., -6.6123e-01,\n",
       "           1.1034e-03,  3.3218e-01],\n",
       "         ...,\n",
       "         [ 1.0358e-01,  6.0011e-01, -9.0201e-02,  ..., -7.2555e-01,\n",
       "           1.6162e-02,  1.8081e-01],\n",
       "         [ 2.1740e-01,  4.2827e-01, -4.8327e-01,  ..., -7.1570e-01,\n",
       "           1.8589e-01,  4.1076e-01],\n",
       "         [ 4.5122e-01, -2.6610e-02,  2.7729e-01,  ..., -8.6081e-02,\n",
       "          -3.5541e-01, -4.7380e-01]],\n",
       "\n",
       "        [[-3.0020e-02, -9.0364e-02, -4.0380e-01,  ..., -1.3385e-01,\n",
       "          -1.9166e-01,  1.1366e-01],\n",
       "         [ 4.7237e-01,  1.1594e-01, -6.9004e-01,  ...,  8.8513e-02,\n",
       "          -4.8052e-01, -1.6785e-01],\n",
       "         [ 6.2622e-02, -4.6986e-02, -6.3620e-01,  ..., -6.1821e-02,\n",
       "          -7.8733e-02,  3.7722e-01],\n",
       "         ...,\n",
       "         [ 5.2156e-01,  2.4339e-01, -8.4875e-01,  ...,  1.7226e-01,\n",
       "          -7.2456e-01, -4.7883e-01],\n",
       "         [ 1.7670e-01, -4.3347e-01, -6.2026e-01,  ..., -9.1872e-02,\n",
       "          -2.8832e-01, -4.2013e-01],\n",
       "         [ 1.5738e-01,  4.4541e-02,  5.2682e-01,  ..., -1.9846e-01,\n",
       "           4.9855e-03, -6.5512e-01]],\n",
       "\n",
       "        [[-7.0171e-02,  1.4629e-01, -6.5545e-04,  ..., -1.3315e-01,\n",
       "          -1.8557e-01, -1.4506e-01],\n",
       "         [-5.6798e-01,  1.1358e-01,  2.6203e-01,  ..., -3.1417e-01,\n",
       "          -2.2627e-01,  4.6190e-01],\n",
       "         [-4.5758e-01,  2.3964e-01,  2.7202e-01,  ..., -3.4917e-01,\n",
       "           5.1958e-01,  2.3823e-01],\n",
       "         ...,\n",
       "         [ 6.2498e-01,  4.8553e-01,  7.6262e-02,  ..., -1.0868e-01,\n",
       "          -6.6887e-01,  3.4586e-01],\n",
       "         [-1.3096e-01,  2.8419e-03, -1.2460e-01,  ..., -1.2445e-01,\n",
       "          -1.3163e-01,  4.6929e-02],\n",
       "         [ 1.2285e-01,  3.7550e-01,  3.2030e-02,  ..., -1.9525e-01,\n",
       "          -2.4123e-01, -8.5524e-02]]], grad_fn=<BmmBackward0>)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m在当前单元格或上一个单元格中执行代码时 Kernel 崩溃。\n",
      "\u001b[1;31m请查看单元格中的代码，以确定故障的可能原因。\n",
      "\u001b[1;31m单击<a href='https://aka.ms/vscodeJupyterKernelCrash'>此处</a>了解详细信息。\n",
      "\u001b[1;31m有关更多详细信息，请查看 Jupyter <a href='command:jupyter.viewOutput'>log</a>。"
     ]
    }
   ],
   "source": [
    "#上面自注意机制网络，请编程生成一个简单的数据集，并使用这个网络进行前向传播\n",
    "x = torch.randn(10, 20, 30)\n",
    "model = SelfAttention(30, 40)\n",
    "output = model(x)\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch24",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
